Test Pyramid, a Discussion and a Practical example
It is hard to find someone who is involved in the software delivery process that has not heard about the Testing Pyramid. Unfortunately though, we see many organizations fall into traps when attempting to implement a Testing Pyramid strategy resulting in a lack of good QA practices. The traps are many and varied. For example, a fast-growing startup receives lots of investments, struggles with hard deadlines resulting in a misunderstanding of how the Testing Pyramid strategy should be applied. Or take the case of a well-established software development team that is stuck in the old way of doing things and just can't seem to find the time to modernize the legacy methods and practices that have become a way of life.
So, what should software development teams do if they want to leverage a Testing Pyramid strategy and integrate optimal QA practices into their software development processes? Here are three steps that they should pay attention to:
- Automation of tests
- Software delivery pipelines
- Build layers of tests to achieve maximum output and velocity.
Automation testing increases the overall productivity of the team and significantly decreases feedback times for newly developed features. Thus, test automation helps reduce the feedback cycle and brings faster validation to the software development process. Automated Testing is also fundamentally important for and effective CI/CD Pipeline. One of the most compelling features of a continuous integration/continuous delivery (CI/CD) solution (also called a DevOps pipeline) is the opportunity to test more frequently without burdening developers or operators with more manual work. Therefore, automation testing is a vital component in the implementation of continuous delivery practices, where the software gets validated automatically, depending on the events that trigger exact types of auto tests.
These various types of tests are usually divided by levels as described below:
Low Level Tests: These tests run against low level functionality of the software platform. Though the number of tests is usually large, the execution time is usually low as each test is narrow in scope and complexity. Low Level Tests are crucial in testing the individual methods and functions of the underlying classes, components and modules (back end) of the software platform. A simple example of a low-level UI test can be a login page where the test is to verify all boxes, shapes, colors, font size, visibility as per specifications, etc.
Higher Level Tests: These tests define the functionality of the software in a broader way without going into deep functionality. The execution of such tests gives a chance to explore more edge cases while also increasing the chance of finding new bugs.
One can imagine that manually doing all the testing can be challenging to say the very least. Building, testing and deploying an ever-increasing amount of software manually soon becomes impossible — without automation too much time is spent with manual, repetitive work instead of delivering working software. Only one solution remains – “Automating everything — from building to tests, deployment and infrastructure — is the only way forward”. So, this statement actually describes the common Test Pyramid.
Automation of testing is key for continuous delivery of software. In order to increase the pace of innovation software development teams must deliver faster without sacrificing quality.
If a company already has automated tests that are executed as part of their pipelines it is not a trivial task to optimize them to decrease feedback time. Analyzing test automation results will help organizations speed up the debugging process and handle known issues more efficiently. In analysis of these automation tests, the team can bring good value by analysis of overlaps. These overlaps will allow to level down such tests and disable their execution on the higher level. More details about automation tests, their granularity, philosophy and information about delivery pipelines will be covered a bit later in the next series of articles.
Let us conclude though with one practical example.
One of the product team at ISG needed to develop a browser based application (front end) which interacted with a set of REST based APIs (back end). The UI supported functionality which generated reports, depending on pre-selected parameters and the end user to download such reports as files to local storage. To test this functionality UI tests were written that generated such reports, produced downloadable files and validated the content of these files. There are lots of parameters that can be present for these reports which resulted in long test execution times, thus increasing the feedback loop. To align this example with the Test Pyramid principle we left only one UI test which validated UI functionality so that users can download non-empty reports (the happy path). All other test variations and test of various combinations of parameters were moved to low level tests; API backend tests. API tests usually run much faster because they are more narrow and targeted in scope. Making this shift resulted in drastically reduced feedback time.
At ISG we have successfully optimized the performance of automated testing through fine-tuned adjustments and adoption of Test Pyramid concepts. We leverage different types of automated tests as a way to deliver success with a balanced approach. The Test Pyramid is a strategy that can help you better balance your tests, reduce test execution time, reduce cycle times, reduce the cost of changing the functionality of your applications, and in general help you move toward a more Optimized Software Development process.
Written by: Volodymyr
