Smoke testing vs. sanity testing explainer on key differences
While smoke and sanity testing are both techniques QA professionals use to quickly identify feature defects, they differ when it comes to scope, automation requirements and timing.
Smoke testing and sanity testing are two QA techniques that aim to find software defects as quickly as possible. Detecting these issues early can eliminate the need for repetitive testing and post-deployment application rebuilds. Essentially, successful smoke and sanity testing can make the overall testing process more efficient and cost-effective.
Many similarities exist between smoke and sanity testing, but it's still important for testing teams to understand the two testing types' differences. Specifically, teams should understand each technique's underlying execution requirements, like the differing environments the tests run in and types of team members involved.
What is smoke testing?
Smoke testing is the type of test that ensures a build's stability. It flushes out any high-severity, showstopper defects that prevent tests from continuing. The results of a smoke test determine whether a build meets quality-gate criteria. If it does meet the criteria, QA can accept the build into the environment and proceed with further testing. Under this approach, smoke testing is a form of acceptance testing.
[Smoke testing] flushes out any high-severity, showstopper defects that prevent tests from continuing.
Testing teams can execute smoke tests in the development environment alongside component integration, through QA and user acceptance testing (UAT), and -- in some cases -- in production environments. In DevOps environments, smoke testing is automated and performed continuously as a part of the continuous integration process.
While testing teams are typically the ones who execute smoke testing, developers can use these tests during builds to make sure components merge properly. In IT organizations that practice test-driven development, testers and developers can work together on smoke tests.
To create a smoke test suite, teams should target test cases that contain critical application functionality. Because the smoke test suite will run frequently, teams should consider investing in test automation. Automating the smoke test suite increases testing efficiency and allows for a more comprehensive test than would be possible for QA professionals to perform manually.
That said, when a build contains complex code, it is a best practice to perform a few manual tests on that code specifically. During QA, teams include a few manual UI tests to ensure that there are no high-severity user experience issues. Such manual UI tests are particularly important in IT organizations that practice continuous deployment.
What is sanity testing?
Sanity tests are high-level assessments designed to ensure an application's reliability. Specifically, this testing type focuses on test cases that contain new features to identify defects in the build. Often, sanity tests also include a test of critical application workflows and integrations. Testing teams will often perform sanity testing when there isn't enough time to attain full test coverage, especially when it comes to UI, cross-browser and mobile components.
Specifically, [sanity testing] focuses on test cases that contain new features to identify defects in the build.
Agile development shops can use sanity testing as a type of mid-sprint QA. Because sprints typically start and finish quickly, teams won't usually have time to perform comprehensive regression testing. In a way, sanity testing can provide an alternative to regression testing, as the technique takes a narrow look at application features to identify deep-seated defects.
Sanity testing typically takes place in a staging environment. These tests can also occur in UAT environments when development teams are under pressure to push remediations for defects that are present in a production environment.
Smoke testing vs sanity testing broken down
Comparing smoke and sanity testing
The key differences between smoke and sanity testing come down to three major variables:
Scope of each testing type's coverage
Role of automation
When these tests occur
Essentially, smoke tests verify stability of the build, while sanity tests concentrate on specific features and defects within the build. Because smoke tests verify the entire application at a high level, they are often seen as a part of acceptance testing. In that same vein, some QA professionals consider sanity tests to be a limited version of regression testing. After all, sanity tests focus on specific features and functions of a specific application.
When it comes to automation, smoke testing is almost always scripted. Sanity testing, however, is primarily an unscripted process because it must target very specific application functions and features. Although testing teams can automate either type of test, sanity testing often uses manual test cases in order to review particularly complex collections of code.
Finally, smoke and sanity testing require different timing and scheduling. Organizations that practice Agile and DevOps methodologies will need to perform smoke testing early and often throughout development. However, once the application is built, teams can execute sanity testing to verify the functionality of new features and look for any production-level defects to fix.
