How much do you test your applications?
When you decided to test, how do you decide what to test? Many people answer with “We must test everything!” Have you ever thought of doing that—testing EVERYTHING?
How long would it take for one tester to test JUST the front end of an application with 10 data entry screens, each with 10 fields that accept two data types? Remember, you also need to accomplish positive and negative testing. If one field will take up to ten characters, then you will need to test one, two, and then three all the way up to ten characters - for EACH field, on EACH screen. Also, there are the combinations between the fields. How long will it take?
Take some time and rationalize what is tested. Testing usually occurs under a timeline that is too short for testing to be thorough as you would like. Rationalization must allow you to thoroughly test as much of the application as needed. Risk assessment is one way to assist in test case rationalization.
Determine the risk
The answer to the original question, “How much do you test?” will probably come from the person who asked the question. The answer is “What is the risk?” Risk can be defined as the probability of an issue occurring and the impact of that issue. You shouldn’t test simply because testing is in the project plan. Testing is in the project plan to reduce risks.
Sometimes risk is beyond the software under testing. The risk may cause the company to lose credibility or stock prices to go down. Project risks may be elevated due to regulations or other legal binding. Discover those risks and reduce them as much as possible, within the project’s constraints (time and budget).
Risk can help prioritize test cases. Ask yourself these questions to determine priority. Which tests cases use the most common pieces of the application or are performed on the most complex parts of the application? Processes that are most used and/or most important to the end users are typically higher priority. Tests that are likely to find the most defects will also be a higher priority. Time spent studying the basis for testing, requirements, user manuals, business processes, etc. can be essential in determining test case priority. Keep in mind that priorities can change each time the application is tested. Therefore, each regression test cycle may not be exactly the same. If the priorities change, so do the test cases.
Delivering accurate information to the decision makers on the project is one of the most important responsibilities of a tester. Making sure that the information is not only accurate, but meaningful, is the hard part. Stating that the testers have found 100 defects is not the entire story. How those defects relate to the identified risks. Knowing the risks, test leads can assist in answering the question, “Should we keep testing?” They should also be constantly reviewing and evaluating the defined exit criteria to answer the question.
If this was a magical world and testers were given unlimited time, they would be able to test an entire application. (Of course that assumes that you don’t get bored to death testing the same application for your entire career!) Testing an entire application is just not feasible.
The answer to how much testing is enough will be different according to :
- The industry
- The availability of time
- Budget
- Testing resources
- Experienced users
Most importantly, ensure the risks are defined in the test plan and are re-evaluated often!
I would love to know what you think. Do you wish there was an unlimited amount of time for testing? Do you think testing 100 percent is even a feasible option? Share your thoughts with me and other readers in the comments section below.
Written by
Sarah Roderus,
Vice President, Consulting Services
TCT Computing Group, Inc.
