Investing in Software Testing: The Cost of Software Quality
Rex Black,
President and Principal Consultant of Rex Black Consulting Services, Inc.
Table of Contents
What Does Quality Cost?
A Hypothetical Case Study
Is This For Real and How Do We Get There?
The process of negotiating a software testing budget can be painful. Some project managers view testing as a necessary evil that occurs at the end of the project. In these people's minds, testing costs too much, takes too long, doesn't help them build the product, and can create hostility between the test team and the rest of the development organization. No wonder people who view testing this way spend as little as possible on it.
Other project managers, though, are inclined to spend more on testing. Why? Smart software managers understand that testing is an investment in quality. Out of the overall project budget, the project managers set aside some money for assessing the product and resolving the bugs that the testers find. Smart test managers have learned how to manage that investment wisely. In such circumstances, the test investment produces a positive return, fits within the overall project schedule, has quantifiable findings, and is seen as a definite contributor to the project (see Bullock, 2000).
What Does Quality Cost?
Phillip Crosby tells us that "quality is free" (Crosby, 1979). Crosby, Campenella, and others who have studied the field of quality costs start by breaking down these costs as follows:
Cquality=Cconformance+Cnonconformance
Conformance costs include prevention costs and appraisal costs. Prevention costs are moneys spent on quality assurance—tasks like code reviews, walkthroughs, or inspections, requirements definition, and other activities that promote good software. Appraisal costs include moneys spent planning test activities, developing test cases and data, and executing those cases—once.
Nonconformance costs come in two flavors, internal failures and external failures. The costs of internal failure include all expenses that arise when test cases fail the first time they're run—as they often do. Think through the process: The tester researches and reports the failure, the developer finds and fixes the fault, the build engineer produces a new release, and the tester retests the new release to confirm the fix and to check for regression. The costs of external failure are those incurred when, rather than a tester finding a bug, the customer does. In these cases, not only does the same process described above occur, but you also incur the technical support overhead and the more expensive process of releasing a fix to the field rather than to the test lab. In addition, consider the intangible costs: Angry customers, damage to the company image, lost business, and maybe even lawsuits.
Two observations lay the foundation for the enlightened view of testing as an investment. First, like any cost equation in business, we will want to minimize the cost of quality. Second, while it is often cheaper to prevent problems than to repair them, if we must repair problems, internal failures cost less than external failures.
A Hypothetical Case Study
Assume we have a software product in the field, with one new release every quarter. On average, each release contains 1,000 "must-fix" bugs—unacceptable defects—which we identify and repair over the life of the release. Currently, developers find and fix 250 of those bugs during development, while the customers find the rest. Cost of quality accounting shows that, on average, each pre-release defect costs $10, while field failures cost $1,000. As shown in the "No Formal Testing" column in the figure, our cost of quality is three-quarters of a million dollars—and customers are mad! So, we invest $70,000 per quarterly release in a manual testing process. The "Manual Testing" column shows how profitable this investment is. The testers find 350 bugs before the release, which cuts almost in half the number of bugs found by customers. Not only does this make the customers happier, but, because cost of quality accounting shows that we pay only $100 on average for each bug found by testers, our total cost of quality has dropped to about half a million dollars. Our ROI on this $70,000 investment is 350%. (For information on bug costs, see: Fewster and Graham, 1999; and Kan, 1995.)
In some cases, we can do even better. For example, suppose that we invest $150,000 in test automation tools, amortizing that investment over twelve quarterly releases, and manage to find about 40% more bugs? Finding 500 bugs in the test process would lower the overall customer bug find count for each release to 250—a huge improvement over the initial situation. In addition, cost of quality would fall to a little under $400,000, a 445% return on investment.
Is This For Real and How Do We Get There?
Cost of quality analyses on software process improvement bear out these figures. Case studies describe reductions in the cost of software quality from 70 percent of the total production cost to twenty to thirty percent (Campanella, 1999). While testing is only part of the investment, it is an important part, and a substantial investment is justifiable to achieve such phenomenal gains.
To get started, you'll need a management team wise enough to look at the cost of quality over the entire life of the software release. A management team that ignores the long term and focuses just on the budget required to get the software out the door initially does not see testing as an investment. Quality is given lip-service when the only priorities are shipping something—anything—on a given schedule and within a given budget.
However, having supportive, far-sighted management is only a necessary—not a sufficient—condition for achieving positive returns on your test investment. Just as in the stock market, there are right and wrong ways to invest. Picking the right tests, managing the appropriate quality risks, using the proper tools and techniques, and driving testing throughout the organization will result in optimal returns, while failure in any one of these areas can mean disappointing or even negative returns. In the next few columns, I'll explore each of these topics.
Author Biography
Rex Black (rex_black@rexblackconsulting.com) is the President and Principal Consultant of Rex Black Consulting Services, Inc. (www.rexblackconsulting.com) an international software and hardware testing and quality assurance consultancy. He and his consulting associates help their clients with implementation, consulting, training, and staffing for testing and quality assurance projects. His book, Managing the Testing Process, was published in June 1999, by Microsoft Press.
Bibliography
J. Bullock, Calculating the Value of Testing, Software Testing and Quality Engineering, Volume 2, Issue 3, 56-62 (May/June, 2000).
P. Crosby, Quality is Free. McGraw-Hill, New York, 1979.
J. Campanella, ed., Principles of Quality Costs. ASQ Quality Press, Milwaukee, 1999.
M. Fewster and D. Graham, Software Test Automation. Harlow, Addison-Wesley, 1999.
S. Kan, Metrics and Models in Software Quality Engineering. Addison-Wesley, Reading, 1995.