Exploratory testing in an Agile environment – Part 1 of 3

This post was written by Noy Bar from the Customer Success team.

 

This is the first in a three part weekly series on exploratory testing in an Agile environment.  In this first part, we discuss what exploratory testing is, where it came from, and its characteristics.  The second part will look at some of the practical aspects of exploratory testing, and the last part discusses when exploratory testing should be used.

 

Background

Software testing began as a way to measure software quality and usability before it reaches its target audience. Over time, the investment in software testing has become at least as much as the investment in coding. .  Despite this huge investment in testing, major companies have still had to recall products and even compensate customers due to poor quality – an expensive outcome.  For example, Microsoft received so many complaints about the quality, performance, and security of Windows Vista, that it was forced to extend support for Windows XP.  

 

The definition of software testing varies depending on whom you ask.  But in essence, software testing is the activity of processing the software (function, feature or other capability) with the goal of identifying behavior that was neither designed nor expected.

 

The most common approach to software relies on documented scenarios. The tester executes a set of written steps against the software (or runs a script to do it for them), then compares the actual result with the expected result. If the results don’t match, it’s reported as a defect. 

 

This is an effective way to test because:

  • Tests can be executed again (and by different testers) by following the documentation
  • Progress is quickly determined by comparing how many tests have been executed to the total that need to be run
  • Well documented scenarios  can be reviewed and approved by stakeholders such as business analysts, developers, product managers and customers
  • Running scripts requires only basic skills, allowing non-technical testers to execute and evaluate the results
  • Well documented scenarios leads to rigorous planning of scripts, which can be tuned to a specific scenario

This technique is useful, but it has a few flaws - particularly after the first release of the software.  Subsequent releases usually require more complex environments and larger testing suites to support both the existing and new functionality.  Consequently:

  • There may be a large number of test cases, which significantly increases the time to execute them
  • Repeatedly testing software with the same tests causes the system to become immune to these tests
  • Maintenance of a large number of tests requires a lot of effort and a deep understanding of how each change to the software affects the tests
  • Written scripts cause the testers to stick to the text instead of being innovative and creative while testing. This of course also has an effect on the tester’s motivation and mood
  • Documenting complex, multi-function, or highly algorithmic processes is difficult, particularly when the expected result cannot be predicted.  This makes it hard for tests to reflect the end-user experience
  • Since testing scenarios are based on system requirements, it is difficult to identify areas where requirements are missing or incomplete - increasing the risk of defects going undetected
  • It is difficult, and sometimes impossible, to provide a concise overall view of software quality during the testing stage

As software development practices move from sequential development models such as Waterfall, to iterative models such as Agile or Lean, the QA department must provide quicker, more accurate status udates on an increasingly frequent basis. The ability to provide updates by traditional scenario-based testing is limited, so we need a solution to support this fast-moving new reality.

 

Exploratory testing

Exploratory testing was born out of ad hoc testing, which is seen as unstructured, sloppy and careless testing performed by unskilled testers, without any planning or documentation.  The term ‘exploratory testing’ was coined to emphasize the fact that structured thought processes are involved, differentiating it from ad hoc testing. Exploratory testing today is commonly defined as simultaneous learning, test design, and test execution.

 

The essence of exploratory testing is a free form procedure by which the tester simultaneously explores and learns the system while testing it.   Although it is free-form, there is still systematic structure, which leads to repeatable and measurable outcomes.  This is an important complementary activity to other testing methods.

 

James Bach likens exploratory testing’s process of scanning the system, trying things out and looking for patterns to solving a jigsaw puzzle. Testers  need to understand the scope of the tests (ie. the picture) and then decide on a testing strategy which results in all the tests (ie. the pieces) fitting together to cover the whole application that is being tested.  As the testing proceeds, the steps that were taken are documented.

 

Characteristics of exploratory testing

These are the main characteristics of exploratory testing as described in a paper by Juha Itkonen and Kristian Rautiainen:

  1. Exploratory testing involves exploration without using a predefined sequence of steps
  2. The exploratory tester is steered by the results of previous tests and the experience gained from them, as well as available documentation such as requirements, guides, market research, etc.
  3. The aim of exploratory testing is to find defects now - not to create tests to execute later
  4. In an exploratory testing session, the tester simultaneously learns the system under test, designs the tests, and runs them
  5. The tester’s knowledge, skills and experience determine the effectiveness of exploratory testing

In addition, Bach notes that tests and test data can be updated as a result of feedback from exploratory testing, which gives the tester greater freedom of expression than in other testing methods

 

 

 

Next week, in the next post in this series, we’ll investigate some of the practical aspects of exploratory testing.

 

Leave us a comment in the box below to let us know if you utilize exploratory testing in your organization.

 

Thanks to Noy for providing this post!

 

 

Comments
SSE(anon) | ‎08-22-2013 06:09 AM

Why can't we use a structred test case execution and exploratory testing simultaneously?

 

Yes its true the exploratory testing truly depends on the experience and depth of the knowledge in the domain. But simultaneously if the structured cases are also used along with exploratry testing atleast high priority test cases (of structred casest), then there is a good chance to identify critical issues at the earliest. This will give the developers and other stake holders a fair idea where the stability of the developed pts is standing.

 

I am senior software test engineer and I have around 7 yrs exp, and I had tried this approach several times. Its like 50% structured document and 50 % exploratory testing which constitute the pdt or the software stability, there after cosmetics issues, usability issues can be identified and that can also be solved. This will help the respective stakeholder like developers Business analysts to priortise the issues and later solve the same accordingly. Business Analysts will get an idea of the missing req and can be cross checked with the customer also.

 

So that at final release to the customer, atleast 80 to 90% the software will be free of issues with respect to the requirment.

| ‎08-22-2013 09:28 AM

Thank you for your comment.

 

In this article, or at all, we never approach Exploratory Testing as a unique and only way to do testing. Specially in the Agile world, additional automated testing such as Dev testing, Unit testing, group testing, acceptance testing etc. Are happening and it is good they do- because identifying problems in an early stage is critical- remember Be Productive, Efficient, Response fast to changes and always be open to feedback.

 

Exploratory testing is complimenting all other testing available in any method, and specially Agile, since it brings in additional perspective which is structured to compare with the others. With that being said, Exploratory testing nor Structured testing are not exclusive.

 

The breakout you present of 50% structured and 50% exploratory does not account to automated testing. It might be more accurate to say that the manual (i.e. not automated) testing can be broken down to 50% structured, 50% automated.

saranya Venkat(anon) | ‎08-28-2013 05:56 AM

Hi Noy Bar,

 

Your blog was really useful. 

 

We have used this method in our project due to time constraint and unprecise requirements. We followed the same procedure of writing the test steps after the exploratory testing and  it immensely helped us to detect more defects. We had the thought that this was not a procedural way of testing . But after seeing your blog, I am assured that this method can definitely be used in the suitable areas.

 

Looking forward to learn more on this topic and also have a question whether this should be used only in Agile method and if it can be used in other methods ?

saranya Venkat(anon) | ‎08-28-2013 06:13 AM

Hi Noy Bar,

Your blog was really useful.

We have used this method in our project due to time constraint and unprecise requirements. We followed the same procedure of writing the test steps after the exploratory testing and it immensely helped us to detect more defects. We had the thought that this was not a procedural way of testing . But after seeing your blog, I am assured that this method can definitely be used in the suitable areas.

Looking forward to learn more on this topic and also have a question whether this should be used only in Agile method and if it can be used in other methods ?

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
Malcolm is a functional architect, focusing on best practices and methodologies across the software development lifecycle.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation