smoke testing
What is smoke testing?
Smoke testing, also called build verification testing or confidence testing, is a software testing method that is used to determine if a new software build is ready for the next testing phase. This testing method determines if the most crucial functions of a program work but does not delve into finer details.
As a preliminary check of software, smoke testing finds basic and critical issues in an application before more in-depth testing is done. If a piece of software passes a smoke test, quality assurance (QA) teams then proceed with further testing. A failure means the software must be handed back to the development team.
The goal of smoke testing is to discover simple but severe failures using test cases that cover the most important functionalities of a software. Smoke tests are performed by QA teams using a minimal set of tests on each build that focuses on software functionality.
Without smoke testing, major issues could slip through the cracks, leaving the chance for those issues to cause larger problems down the line.
The term smoke testing originates from a basic type of hardware testing in which a device passes the test if it doesn't catch fire the first time it turns on. Daily build and smoke tests are among industry best practices advocated by the Institute of Electrical and Electronics Engineers in Code Complete by former IEEE editor in chief and software engineer expert Steve McConnell.
What are the different types of smoke testing?
Types of smoke testing include the following:
- Manual smoke testing. Human software testers conduct smoke tests manually. This includes manually developing and updating test cases. Test scripts are also written manually for new or existing features.
- Automated smoke testing. Software tools are used to automate the smoke testing process. Smoke testing tools make the testing process more efficient by automatically providing relevant tests.
- Hybrid smoke testing. Hybrid testing is a combination of manual and automated smoke testing, where testers write test cases and then automate the tests using a tool.
How smoke testing works and how to perform a smoke test
QA testers perform smoke testing after the developers deliver a new build of an application. The test is performed manually or automatically using previously written scripts.
If the smoke test passes, the software is integrated with an existing build in the QA and staging environment. The software build then moves on to more rigorous tests, such as unit and integration tests. If the smoke test fails, then the testers have discovered a major flaw that halts all further tests. QA teams then ask developers to send a fixed build.
This broad initial test is a more effective strategy to improve software code than if the team conducted specific and rigorous tests this early in the software development process.
Smoke testing is also performed from the perspective of the user experience. This approach includes testing key functionalities, such as if the build is accessible or if the user interface and login mechanism function correctly. Other key functionalities include if a selection action correlates with the intended action.
How smoke tests are run depends on the type of smoke test, and approaches may vary from one organization to another.
A general process for performing a smoke test may look like this:
- Decide between manual, automated or hybrid testing. Manual testing is most commonly chosen first, while automated tests may be done once more elements are added to the smoke test.
- Create testing scenarios. QA testers need to know how many test cases they need for each core software feature. Pass or fail metrics should be decided on based on software and organizational needs and standards.
- Create the smoke tests. Either using manual or automated processes, write and create the test suites.
- Run and record tests. While smoke tests are run, testers should have a process set to record the results for each test. This can either be manual or automated.
- Analyze smoke test results. If the software fails, it's returned to the development team for more testing. If it passes, it is ready for more rigorous functional testing.
Manual smoke tests require the testers to write and update smoke tests. Scripts are manually modified to fit each requirement. In automated smoke tests, additional tools are used to test core features quickly. Hybrid smoke tests have a tester write the test cases, while automating the tests using smoke testing automated tools.
Advantages of smoke testing
Smoke testing is a valuable part of the testing process for the following reasons:
- Automation enables tests to be run faster and more often.
- Future builds are constructed on bug-free software.
- It identifies defects in early stages of development.
- It lessens risks associated with adding code to existing build integrations.
- Smoke tests can be run often due to their low resource requirements and simple process.
- There is flexibility in implementation with manual, automated and hybrid types.
- It verifies software quality early on in development, ensuring that a build is stable.
Disadvantages of smoke testing
There are some disadvantages that come with smoke testing, however, for example:
- Smoke testing doesn't cover all functionalities in a piece of software -- only specified core functionalities are tested.
- Not every bug or problem may be found with smoke tests, and some may still be found later in development.
- Manual smoke testing takes longer to complete when used in larger projects.
What is the smoke testing cycle?
The smoke testing cycle starts with a build being delivered from the development team. That build is passed for an initial smoke test by QA. Specific test cases are prioritized based on core or key features, tests are created and the software is then tested.
If the software fails the initial smoke test, the software is handed back to the development team, where it is fixed and then sent back to QA. Another set of smoke tests is performed on the software, and if it passes, the software is integrated with an existing build in the QA and staging environment. The software is now ready for more rigorous functional testing.
This cycle helps identify basic and critical issues in an application before the team working on it performs more in-depth testing. This is why smoke testing is performed at the initial phase of the testing lifecycle.
Smoke testing examples
Smoke testing tests the basic functionalities and core features of software in development. An example of a smoke test could be the testing of a login function of a web application. Performed from the perspective of the user, the page would be launched, and the tester would enter a valid username and password and attempt to sign in. A successful test would result in a successful login. If the feature doesn't work, it is sent back to the development team.
Another example of a smoke test could be the testing of if an item can be added to a cart in an e-shop. Again, a tester would launch a specified page and attempt to add the item to their cart. A successful test would see the item added to the cart, while a failure would result in an item not getting added to the cart.
Another feature, such as the search function in an application, could also be smoke-tested. The application would be opened up, and the user would then navigate to the search bar and search for and select a file to see if they can access it.
Automation and smoke testing
Automated smoke testing can drastically cut down on testing time. Manual testing can take half a day, depending on the number of smoke tests involved. After automating those tests, smoke testing can take only a few minutes. If developers make builds frequently or implement continuous testing, then automating smoke tests enables QA to give faster feedback on test builds.
Tools for automated smoke testing include Selenium and PhantomJS. Selenium is Open Source software that automates and runs testing parameters on multiple web browsers. Selenium automates control of browsers on various operating systems as well. PhantomJS is an option for integration and automation with continuous integration tools, such as Jenkins and TeamCity. PhantomJS is not resource-intensive.
Smoke testing versus regression testing
Regression testing is another way to ensure new code changes won't adversely affect an application's existing codebase. Specifically, regression testing is a type of software test that assesses if changes to an application will introduce defects.
Similar to smoke testing, regression testing is performed by a QA engineer and ensures that changes to a program do not add new bugs. Like smoke tests, regression tests should be implemented often with every new build. Both testing types can also be automated or performed manually.
Unlike smoke tests, which are surface-level tests, regression tests are in-depth and detailed. Regression tests target specific operations of software, taking a few hours to complete.
For example, regression testing explicitly checks each link on an updated webpage to verify that each link still works as intended. While smoke tests focus on verifying a feature's stability, regression tests verify the consistency of a function.
Regression tests are also not responsible for accepting or rejecting software builds like smoke testing is.
Smoke testing vs. sanity testing
Sanity tests are high-level assessments designed to ensure an application's reliability. This testing type focuses on test cases that have new features in order to identify defects in the build. Often, sanity tests also include a test of critical application workflows and integrations.
Both sanity and smoke tests are similar in nature and can be confused. Smoke testing focuses on verifying critical software functionalities before more in-depth testing is performed, whereas sanity testing verifies the functionalities like bug fixes after the build. In other words, while smoke testing is done to verify the stability of a build, sanity tests are done on stable builds. Sanity testing is also a subset of regression testing, while smoke testing is a subset of acceptance testing.
Smoke testing is also typically documented, while sanity testing is not. In addition, smoke testing can be done by either developers or QA testers, while sanity tests are only carried out by the testers.
Learn more about the top software testing methodologies.
