Exploratory testing in an Agile environment – Part 2 of 3

This post was written by Noy Bar from the Customer Success team.

 

In the first part of this three part weekly series on exploratory testing in an Agile environment, we discussed what exploratory testing is, where it came from, and its characteristics.  In this part, we examine some of the practical aspects of exploratory testing, and the last part will discuss when exploratory testing should be used.

 

Practical Aspects of Exploratory Testing

 

PracticalAspects.png

There are five main elements of exploratory testing:

  1. Investigation of the software to be tested
  2. Planning of the tests
  3. Execution of the tests
  4. Testing guidelines
  5. Quantifiable results

Jonathan Bach, in his paper Session-Based Test Management, embodies these five elements in three main activities:

  • Test design and execution
  • Bug investigation and reporting
  • Session setup (in order to perform the above actitvies)

Bach refers to these three activities as "TBS", after their initials.  Let's look at them in more detail.

 

Test Design and Execution

Exploratory testing focuses on the question “What should I test now?”  To answer this question, the tester needs to analyze the Application Under Test (AUT) to understand its purpose and capabilities.  The ability to perform this analysis depends on their technical skills and their understanding of the business.

 

Once the tester understands the AUT, the goals of the testing session are defined, based on 1) overall testing strategy and 2) time available for testing.  Strategy can be affected by:

  • Development progress;
  • Who can assist in the testing;
  • How well the AUT is documented;
  • The technology that underlies the system under test.

A well suited exploratory tester armed with this information and following some basic guidelines, such as a list of functional areas to explore, can then determine how to start, what to focus on, and the scope of each test.

 

Since the tests are not documented and exploratory testing is an unpredictable stream of actions, it’s important to protect and maximize the tester’s focus. This extreme focus gives the tester the unique ability to adapt his or her strategy ‘on-the-fly’ depending on the findings during the test. Executing a test includes interacting with the AUT, observing its behavior and learning while doing it - which is when defects are found.

 

Bug investigation and reporting

Juha Itkonen, Mika V. Mäntylä, and Casper Lassenius reported in their paper Defect Detection Efficiency: Test Case Based vs. Exploratory Testing  (in table 6) that exploratory testing uncovers 11 percent more defects than scenario or test case-based testing. More complex defects, such as usability or business flow issues, can be difficult or even impossible to find using other testing methods. In addition, exploratory testing generates up to two and half times fewer false reports, such as irreproducible and duplicate defects, or a tester misunderstanding the functionality.

 

Session setup

Unlike the other two activities (test design and execution, and defect investigation and reporting) which are fairly self-contained and intuitive, session setup has a wider scope.  It starts before testing begins and accompanies the process of testing throughout.

 

 

It includes everything and anything necessary to enable the other two activities.  For example:

 

  • Preparing any equipment necessary
  • Establishing a testing environment
  • Locating information required to test (for example, usernames and passwords)
  • Reading user guides

The test report also begins at this stage, since it accompanies the the testing.  The report details the faults that were found, questions that arose during the testing, and areas that were covered during the testing session.

 

Measurable results

Any form of testing, including exploratory testing, requires the ability to provide a status and results at any moment. The status should be concise, be accurate enough to reflect the situation, and should not require additional interpretation on the reader’s part.

 

TBS (see earlier in this section) is a basis for metrics that can aid in understanding the status by estimating the amount of time required for each of the activities. The testing activity can be divided into two: planned tests, which are considered prior to the actual testing activities, and opportunistic tests, which are performed as part of the activities. 

 

An AUT that is not intuitive to use might reflect a longer setup time than usual, because it’s difficult for the tester to learn it.  Similarly, a non-intuitive AUT will result in the tester straying from the area which was initially being tested as he or she continues to explore the system.  Although exploratory testing encourages opportunistic testing, it generally focuses on a specific area. If the system continues to lead the tester away from the focus area, there are probably usability issues.

 

In a freestyle exploratory testing session, the status report might contain very little information.  In extreme cases, the only documented outcome of the testing session is the list of defects discovered.

 

The notes that the tester made during the test session, including the list of defects found, should be examined by the test manager, business analyst or customer to ensure that the issues are analyzed and taken care of.

 

Measureable indicators for an exploratory session might include:

  • The number of defects found
  • Questions raised and clarifications required during the test session
  • Number of functional areas covered by the test session
  • Percentage of time required to learn the software relative to the total time of the test session
  • Amount of time invested in investigating defects that were found
  • Amount of time invested in each functional area

A known method to reflect the status of the exploratory testing is to follow a ‘To Do’ list of areas/functionalities that should be tested. In an Agile environment, this would be the list of user stories.  It is important to mark the status of each finished test so that an overall status report can be given at any time.

 

Note that testers who are capable of explaining the logic and strategy of their testing typically produce a better and more accurate status.

 

 

 

Next week, in the final post in this series, we’ll look at when to use exploratory testing.

 

Leave us a comment in the box below to let us know how you do exploratory testing in your organization.

 

Thanks to Noy for providing this post!

 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Malcolm is a functional architect, focusing on best practices and methodologies across the software development lifecycle.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation