51Testing | 17 Years As A Software Testing Service Provider
What is your test-to-development ratio? Do your testers participate in the whole stage? If testers engage in the whole stage, what testing needs to be done to ensure the testing quality?
The automation testing coverage is required to reach 99% including functional test, performance test, and usability test.
The first two questions are the most concerned and asked, while the third one is the practical concern of the project. How do testers carry out tests, and how much the coverage of testing should be? All these questions are related to lean testing.
Lean Production
Before introducing lean testing, let’s know about lean production first.
Lean production is an approach to management that focuses on cutting out waste, whilst ensuring quality. This approach can be applied to all aspects of a business — from design, through production to distribution.
Lean production aims to cut costs by making the business more efficient and responsive to market needs.
This approach sets out to cut out or minimize activities that do not add value to the production process, such as holding of stock, repairing faulty product and unnecessary movement of people and product around the business.
Lean Testing
The purpose of shift left and fully test required by agile testing is for quick feedback and cost reduction in order to deliver high-quality software. If introducing the concept of lean into agile testing, will help to reduce waste during the testing process and maximize the value of testing.
For lean testing, you have to understand that you can not blindly pursue the testing coverage. The high and comprehensive coverage is a waste, the effective testing is more valuable.
Whether it is manual testing or automatic testing, you have to understand the business value and quality mission firstly, and then perform testing according to the business risks. Focus on high priority, and reduce testing coverage in lower priority.
According to the 80/20 rule, 80% of businesses with high priority may just lay on 20% of the function module, while the other 80% of the function module occupies 20% of the function module. If you treat them equally and pursue full coverage, it will cause a lot of waste. On the contrary, if you pursue the testing effectiveness rather than the coverage, the ROI will be higher. So it is important to hit the spot. Launch the software with bugs could be a good strategy.
Note that quality objective is the key. For some software concerning life security, the quality requirements will be particularly high. The comprehensive testing coverage is also effective and it is just the right kind. This is the core concept of lean testing.
Take business value as objective and deliver high-quality software with cost as low as possible. Make the testing embody its value, achieve effective coverage and reduce waste.
The Essence of Lean Testing
According to the definition of lean testing, the quintessence of lean testing can be summarized as TAT: TIME, AMOUNT, TARGET.
TIME
Agile testing requires participation in the entire process and in every link of the software development. To have the right test in the right moment, that is, the concept of “TIME”. For example, the confirmation of the demand before development, the realization of automated testing and verification before the development of software, the smoke testing after deployment, and so on.
AMOUNT
For test coverage, some people may think the higher the better. For example, as we mentioned above, the team requires 99% for automated test coverage, even if this test coverage is effective, it takes too much effort to test some modules that are not so important or some modules that are not prone to problems. It may outweigh the gain and cause waste. It is suggested that the test coverage, whether it is manual or automated, should be at an appropriate amount. And testers should increase the business priority for testing and the test coverage of response according to risks assessment. Blindly pursue high coverage is not a clever choice. Considering the pros and cons, spend your time on truly valuable tasks, that’s the concept of LEAN.
TARGET
Lean testing generally refers to the specific automated testing of the zone influenced by the modification of code. While the lean testing we refer has a wider scope, which can be understood as risk-based testing. Risks may come from business or architecture levels, or the modification of code, or the system features, or other related elements of the project. Before performing testing, analysis and design are much more important.
Guiding Framework of Lean Testing
There are two general guiding frameworks of lean testing: testing quadrants and testing layers.
Testing quadrants
Testing quadrants is firstly proposed by Brian Marick, and Lisa Crispin and Janet Gregory quoted this theory and make a detailed introduction in their book Agile Testing: A Practical Guide for Testers and Agile Teams.
The right and left sides refer to the role of testing which is guide development and critique the products, while the upper and lower sides refer to the test-oriented objects which are business-facing and technology-facing.
- Guide development
The left side is to tell the team what code to write, clarify the requirements and assist the testing design.
The first quadrant is technology facing of guide development helping build internal quality, that is, the assurance of code quality, such as units test and API testing, etc. The second quadrant is business-facing of guide development, confirming the expected system in a way that business experts can understand and helping the team to understand the business and its value from a higher level. The testing in these two quadrants can quickly provide feedback and ensure rapid problem resolution. Not only guide the functions developments but also provide a protecting procedure that prevents unexpected behaviors from modified and new codes.
- Critique the product
The code written by the programmer may satisfy the requirement of a business-facing test on the left, however, it may not be the product that the client really wants. Therefore, the third and fourth quadrants of product critique are necessary.
The third quadrant of product critique is business-facing which simulating the way that real users do with the product to confirm whether it is the product really needed. The fourth quadrant of product critique is technology facing which mainly uses tools and related techniques to evaluate the performance, robustness and security. In every period of development, these tests should be considered.
The information collected from the tests in these two quadrants should be sent to the quadrants of the left side for creating new testing to drive the next development. This will create a positive enhancement loop.
How to use testing quadrants?
The order of quadrants has no relationship with the order of testing execution. Generally, agile testing starts before business-oriented testing. The factors related to the order of testing are as followed:
- The risk of product launching
- The requirement of product from the business part
- Whether it is the development based on an existed system or the development of a new system
- Available testing resources
The testing quadrants only provide a framework to confirm the tests required to ensure quality. Practically speaking, it should be considered with the project to conduct related tests.
Testing layers
The testing layers theory is mainly aimed at automated testing. It is divided into different layers based on the zone that testing covers.
As for the testing layers theory, people may get more familiar with the testing pyramid. It shows the proportion of different types of testing, more units testing in the lower layer and fewer tests in the upper layer, presenting a pyramid structure.
In fact, the proportion of each testing type is different due to the differences of technical structure, system features, quality requirements and team’s ability, not necessarily a pyramid structure, it could be honeycomb or ice-cream structure as shown below.
Honeycomb and ice-cream structure
Regardless of the ratio, test in each layer has its own feature. The lower the test, the closer to the code, the lower the coding cost, the faster the execution speed, the more accurate positioning the problem, but it is farther from the business and cannot reflect the business value. The upper the test, the closer to the business, the more to reflect business value, but it has advantages of instability, slow execution speed, high cost, and difficulty in problem location. In conclusion, it is necessary to weigh the pros and cons and determine the proportion of tests in each layer according to the specific condition and goal of the project.
In conclusion
The lean testing theory mainly helps the team to develop an appropriate strategy, not a specific testing method. The core of lean testing is time, amount and target, which is to make the testing just right to reduce waste.
Test strategy is affected by many factors, needs to be goal-driven, and constantly evolve over time according to specific conditions. The four quadrants of the testing and testing pyramid are only a guiding framework, not a specification that must be followed, and can be used as a reference model for test strategies. The strategies of specific projects need to be adjusted according to the specific conditions of the project.
