Do you have to choose between smoke and regression tests?

Smoke test development and regression test development are related and similar, but with a slightly different angle. The biggest differentiating factor in smoke versus regression tests is the depth and scope of the test -- or how far it goes through the application. Does it cover multiple configurations or a single set? These are the type of questions you need to answer in order to build valuable smoke and functional regression suites. Define what your smoke tests verify first. Are they verifying the application installs and opens as expected, or do they go deeper than that?

Functional regression test cases are developed to test a full function -- or at least a piece of it -- and then recombined to test the application from end to end in various possible scenarios. Granted, most applications contain so many variances in the scenarios it's impossible to test them in a reasonable amount of time, let alone the time that's built into a release schedule. However, select the tests that represent the fullest functional cover and go into more depth than your smoke test suite for functional regression tests.

Let's say that as a quality assurance team lead or manager you have a large suite of functional regression tests but none are defined as smoke tests. If your test cases are not prioritized, then your first step is to prioritize which parts of the application are most critical. From there, you can determine which functional area each test represents. Next, look at defect statistics for the past two to three releases and see which areas those defects were found in. Determine if there are one or more areas that have more defect reports against them. Then determine if any of your existing critical tests cover those areas. Add the most critical tests to the smoke test suite. Keep the number under control and keep in mind you have to allow time for research if the tests fail. The smoke test suite should be under 30 tests for manual execution and no more than 50 for automated execution. Otherwise, add them to the functional regression suite.

It's advantageous to make the smoke and functional testing suites complement each other. In other words, they should both test the critical areas of the application but to varying depths or degrees of detail. The simpler tests need to be developed into a smoke test suite, while the more in-depth tests are added to the functional regression suite. I don't believe in superficial testing within a smoke test suite -- if the only time I have allows me to execute a smoke test suite, I really want to feel I've exercised the application. Superficial tests can easily be covered within functional regression tests. For the smoke test suite, focus on tests that provide immediate value and cover functionality the customer truly cares the most about.