The Four Primary Objectives of Testing
Testing can be applied to a wide range of development projects in a large number of industries. In contrast to the diversity of testing opportunities, there is a common underpinning of objectives. The primary motivation for testing all business development projects is the same: to reduce the risk of unplanned development expense or, worse, the risk of project failure. This development risk can be quantifi ed as some kind of tangible loss such as that of revenue or customers. Some development risks are so large that the company is betting the entire business that the development will be successful. In order to know the size of the risk and the probability of it occurring, a risk assessment is performed. This risk assessment is a series of structured “what if” questions that probe the most likely causes of development failure depending on the type of development and the type of business the development must support. This risk motivation is divided into four interrelated testing objectives.
Testing objective 1: Identify the magnitude and sources of development risk reducible by testing. When a company contemplates a new development project, it prepares a business case that clearly identifi es the expected benefi ts, costs, and risks. If the cost of the project is not recovered within a reasonable time by the benefi ts or is determined to be a bad return on investment, the project is deemed unprofi table and is not authorized to start. No testing is required, unless the business case is tested. If the benefi ts outweigh the costs and the project is considered a good return on investment, the benefi ts are then compared to the risks. It is quite likely that the risks are many times greater than the benefi ts. An additional consideration is the likelihood that the risk will become a real loss. If the risk is high but the likelihood of the risk occurring is very small, then the company typically determines that the risk is worth the potential benefi t of authorizing the project. Again, no testing is required. If the risk is high and the likelihood of its occurrence is high, the questions “Can this risk be reduced by testing?” and “If the risk can be reduced, how much can testing reduce it?” are asked. If the risk factors are well known, quantifi able, and under the control of the project, it is likely that testing can reduce the probability of the risk occurring. Fully controlled tests can be planned and completed. If, on the other hand, the risk factors are not under control of the project or the risk factors are fuzzy (not well known or merely qualitative), then testing does not have a fair chance to reduce the risk.
Testing objective 2: Perform testing to reduce identifi ed risks. As we will see in subsequent chapters, test planning includes positive testing (looking for things that work as required) and negative testing (looking for things that break). The test planning effort emphasizes the risk areas so that the largest possible percentage of the test schedule and effort (both positive testing and negative testing) are dedicated to reducing that risk. Very seldom does testing completely eliminate a risk because there are always more situations to test than time or resources to complete the tests. One hundred percent testing is currently an unrealistic business expectation.
Testing objective 3: Know when testing is completed. Knowing that 100% testing of the development is unachievable, the tester must apply some kind of prioritization to determine when to stop testing. That determination should start with the positive test items in the test plan. The tester must complete the positive testing that validates all the development requirements. Anything less, and the tester is actually introducing business risk into the development process. The tester must then complete as much of the risk-targeted testing as possible relative to a cost and benefi t break-even point. For example, if there is a $10,000 business risk in some aspect of the development, spending $50,000 to reduce that risk is not a good investment. A rule of thumb is a 10–20% cost to benefi t break-even point for testing. If the same $10,000 business risk can be thoroughly tested for $1000–2000, then cost to benefi t is very favorable as a testing investment. Finally, the tester must complete as many of the negative test items in the plan as the testing budget allows after the positive testing and risk testing are completed. Negative testing presents two situations to the test planner:
- The fi rst situation is the complement of the positive test items. For example, if a data fi eld on a screen must accept numeric values from 1 to 999, the values 1, 10, 100, 123, 456, 789, and 999 can be used for positive test completion while the values 1, 0, and 1000 can be used for negative test completion.
- The second situation is the attempt to anticipate novice user actions that are not specifi ed in the requirements or expected during routine business activities. Planning these kinds of tests usually takes deep insight into the business and into the typical ways inexperienced business staff perform routine business activities. The time and expense necessary to test these “outlier” situations often are signifi cantly out of proportion to the likelihood of occurrence or to the magnitude of loss if the problems do occur.
Testing objective 4: Manage testing as a standard project within the development project. All too often, testing is treated as a simple skill that anyone can perform without planning, scheduling, or resources. Because business risk represents real dollar loss, real dollar testing is required to reduce the risk. Real dollar testing means that personnel with testing expertise should be formed into a testing team with access to the management, resources, and schedules necessary to plan and complete the testing. The testing team, as any other business team, can deliver the testing results on time and within budget if the team follows good standard project management practices. The benefi t of this observation is the reassurance that testing does not have to be hit or miss. It can be planned and completed with the confi dence of any other professional project to achieve its objectives. The liability of this observation is the realization that testers are a limited resource. When all available testers are scheduled for an imminent testing project, further testing projects cannot be scheduled until you fi nd additional qualifi ed testers.
