end-to-end testing

What is end-to-end testing?

End-to-end (E2E) testing is a software testing methodology that verifies the working order of a software product in a start-to-finish process. End-to-end testing verifies that all components of a system can run under real-world scenarios.

The goal of this form of testing is to simulate a user experience from start to finish. E2E testing can find software dependencies while also validating the system under test, its data integrity and integrations.

End-to-end testing is typically performed by quality assurance (QA) teams, and are executed in dedicated test environments. This normally takes place after functional and system testing. End-to-end testing starts from the user's perspective, simulating typical operations the application can perform.

Types of end-to-end testing

There are two types of end-to-end testing: horizontal and vertical.

Horizontal end-to-end testing is the most used and well-known approach. Horizontal testing can build confidence in a system by assuming the perspective of a user. Horizontal testing confirms whether a user can navigate through a system, if it works as expected and if there are any unexpected bugs or exceptions.

Vertical end-to-end testing refers to testing in layers, meaning tests happen in a sequential, hierarchal order. Each component of a system or product is tested from start to finish ensuring quality. Vertical testing is frequently used to test critical components of a complex computing system that do not typically involve users or interfaces. A layer without a user interface (UI) would benefit from vertical end-to-end testing over horizontal.

Why use end-to-end testing?

All forms of end-to-end testing simulate real-world scenarios, like the interaction with hardware, networks and databases. End-to-end testing assumes the advantages of white and black box testing.

White box testing closely aligns with vertical end-to-end testing and examines the inner workings of a program or system. Black box testing is comparable to horizontal end-to-end testing and focuses on functionality without examining the internal working structures of a program or system.

The goal of simulating real-world situations from a user perspective helps ensure that software applications are working as they were intended to. It ensures data flow across processes is consistent, discovers system dependencies and validates system integrations.

How to perform end-to-end testing

End-to-end testing is applied differently depending on what software is tested. In an e-commerce web application, for example, horizontal end-to-end testing may include signing into a profile, using search and browse functionalities, adding an item to a cart, saving the item for later, buying an item, logging shipping and credit card information, confirming the purchase and signing out.

By comparison, vertical end-to-end testing would be used in a system where there is no UI, or the UI has a high technical skill cap. For example, it would work to test an application program interface or Structured Query Language interface.

Before testing, designers and developers construct a list of what UI or other functions they need to perform. Data flow should be tracked from system to system to find dependencies, flaws or inaccuracies. Conditions should be prepared before testing, including factors that may change any interface options such as latency or traffic.

End-to-end testing is typically conducted on finished products and systems, which makes each review a test of the completed system and how it communicates with its parts. If the system output is not what is expected, or a problem or issue is found, it is tested a second time. Data is recorded and analyzed to determine the issue's origin, and then it is fixed and re-tested.

Metrics to use in end-to-end testing

Metrics commonly used in end-to-end testing typically include the following:

• Status of test case preparations. This metric tracks the progress of test cases under preparation compared to planned test cases.
• Test progress. This metric tracks details about completed tests in the form of percentages. Tracked percentages include executed and unexecuted tests, passed or failed tests, and valid or invalid tests.
• Status and details of defects. Based on priority and severity, this metric is tracked as a weekly percentage of open and closed defects.
• Availability of the testing environment. This metric keeps track of the operational hours of test environments as scheduled hours of testing per day.

End-to-end testing best practices

Best practices to follow when performing end-to-end practices include the following:

• Conduct a second test. Perform a second test in cases where a problem is found.
• Create a list. Have a list of functions, features and UI elements that should be tested.
• Track data flow. Tracking data flow between systems helps find dependencies or other issues.
• Keep a designated order of testing. Unit and integration tests should be done first to fix the most apparent issues. Each system component should also be E2E tested.
• Keep the setup process simple. It should take little time to revert data in the testing environment.
• Prioritize the user experience. E2E tests should be created with users in mind and should be performed on features that could cause the largest issues.
• Record and analyze data. Recording and analyzing test cases helps determine the origin of an issue.

End-to-end testing vs. other software testing

Software testing involves smaller tests performed at a faster pace -- only relying on methodologies such as end-to-end testing for final integration and pre-release evaluation.

Unit testing

At the bottom of the pyramid, the beginning of development for each system is unit testing. This form of testing addresses smaller pieces of the system or code and is ultimately faster and more efficient than end-to-end testing. If a bug is found in the overall system of an end-to-end test, it could be anywhere in the system. But if a bug is found in a unit test, far less time is needed to find and correct the problem.

Key differences between E2E and unit tests include the following:

• Test scope. Unit tests focus on individual code units, while E2E tests focus on the application from a user's perspective.
• Who performs the test. Developers conduct unit tests, while QA teams conduct E2E tests.
• Automated testing vs. manual testing. Unit tests are commonly automated, while E2E tests can be manual or automated.
• Sequential vs. parallel. E2E tests must be run sequentially, while unit tests can be run in parallel.
• Unit tests may frequently change. This may make E2E testing more time efficient to conduct.

Integration tests

The middle section of the Testing Pyramid is integration, which tests how multiple units operate together with a proper flow of information. Integration testing is similar to vertical end-to-end testing in that it tests the underlying framework of the user interface.

Key differences between E2E and integration tests include:

• Test scope. Integration tests span multiple individual components, while E2E testing tests the entire software stack.
• Test perspective. Integration tests ensure software components work well together, while E2E tests check software from the user's experience.
• Speed. Integration tests are typically faster than E2E tests.

End-to-end tests

The final and top portion of the Testing Pyramid involves end-to-end testing -- also known as UI testing – in which the entire system is tested. If time is spent efficiently, most tests should be unit tests, followed by integration tests and then end-to-end testing.

End-to-end testing vs. system testing

System testing is a process in which QA teams evaluate how the various components of an application interact together in the full, integrated system or application.

System testing verifies that an application performs tasks as designed. With system testing, a QA team gauges if an application meets all its requirements, which include technical, business and functional requirements.

System tests are like end-to-end tests in that they are performed on software as a whole, but they have distinct differences that make them their own form of testing. These differences include the following:

• What the tests validate. E2E validates the software system and interconnected subsystems, while system testing validates software systems.
• What the tests check. E2E tests check complete end-to-end processes, while system tests check system features and functionalities.
• Automated vs. manual testing. E2E tests are more commonly conducted manually because some interfaces might be difficult to automate, while both manual and automated testing can be performed for system tests.

End-to-end testing vs. functional testing

In functional testing, each software function or feature is compared with an organization's specifications to ensure the software provides the output required by the user or business. If a system passes functional testing, it can be verified for release. Functional testing confirms the application meets customer requirements since the process tries to closely reflect the true experience of a user.

End-to-end tests are like functional testing in that they consider the user experience. However, differences between the two include the following:

• Scope. Functional tests validate the input and output result of each test, while E2E tests validate that each step in the process can be completed from start to finish.
• Acceptance. Functional tests ensure software meets established acceptance criteria, while E2E tests ensure a process works after software changes.
• Types. While E2E tests can be horizontal and vertical, there are numerous types of functional testing, including smoke testing, sanity testing, black box testing, integration testing, unit testing and user-acceptance testing.

Benefits of end-to-end testing

Potential benefits to end-to-end testing include the following:

• Detects bugs before release. E2E testing helps catch any lasting bugs before release.
• Expands test coverage. Instead of testing individual software components, E2E tests the software in a start-to-finish process from the user's perspective.
• Helps ensure software performs as expected. Test cases are run as if the user is performing an action.

Challenges of end-to-end testing

For all the strengths that come with end-to-end testing, it does have its challenges. These include the following:

• Speed. Today's software development cycles tend to employ some form of Agile development as a fast-paced and collaborative approach to software development. End-to-end testing may be too slow for Agile methods of software development.
• Designing tests. Designing tests may be difficult because it is hard to gauge how each user will navigate and work the software, making it sometimes difficult to simulate real-world user behavior.
• Workflow. Test cases while using E2E must be run in a set sequence to simulate how users will most likely navigate the software.

Software test engineers can still choose to use end-to-end testing, but the practice can be supplanted by other testing methodologies, such as automated sanity checks.

Learn more about E2E and other software testing methods like unit, integration, system and acceptance tests.