Agile Performance Testing Life-cycle

Customers expect applications to be fast. Studies show that average users expect to wait no longer than two seconds for a website to load. Furthermore, each additional second spent loading a page can cause a 7% loss in customer conversion.

Using Agile methodologies especially while implementing continuous delivery (CD) technique, changes the way we test and improve our application performance. In this blog post and the following one, I describe how to handle performance issues when working in Agile:

  • The Agile performance testing lifecycle - includes the steps performed by performance engineers to deliver complete and timely performance reports.
  • Handling performance bugs in Agile -  explains how developers handle performance testing results.

Part 1 - The Agile Performance Testing Lifecycle

Performance testing in an Agile environment is a unique challenge. Traditionally, performance testing requires precise and long term planning, long cycles and specialized personnel, while Agile development encourages vague long term planning, short cycles, flexibility for changes and self-organizing teams.

Implementing a traditional performance testing lifecycle in an Agile product will deliver poor results. By the time the performance testing report is ready, the application will have changed, and the report will be out of date.

A true Agile performance testing cycle should include the following five phases:

 performance life cycle.jpg

These phases are linked to the product drop cycle, which is the amount of time required to release a new version. These phases may match the Agile drops (or sprints), although it’s not necessary. In some cases the drop cycle may consist of several consecutive Agile iterations.

 

Planning phase

The Planning phase should be executed while planning the next drop. The following main activities in this phase are:

  1. Reviewing the next drop backlog. Performance engineers learn about the features being developed during the next drop, and provide advice about some possible upcoming performance issues.
  2. Setting a time KPI for each new feature. Product owner specify the acceptable amount of wait time the user may experience when using the new feature.

Pre-drop phase

The Pre-drop phase begins a few days before the end of the drop, and helps performance engineers to set up the test environment with a certain amount of expected load for the three months ahead.

Three months is an acceptable timeframe because it is long enough for developers to solve problems found during testing, as well as short enough for product owners to provide fair estimates.

During the Pre-drop phase, product owners provide estimates for the three months ahead, for the following parameters:

  • Data - How much data will the system contain? How many records will there be in major database tables? How big will the disk size be? How many users will the system support? And so on.
  • Flow Which actions and pages will be used? How frequently?

The Pre-drop phase can be executed in parallel with the Regression phase.

 

Regression phase

The Regression phase begins when the current drop is finalized and ready for deployment, and identifies regressions found in the new drop when compared with the previous drop. To verify that the results are a true regression, tests must run with the same data and load the same configurations as was used for the previous drop.

Running regression tests should provide similar (or better) results than the previous version. However, additional time required can be caused by new functionality, and does not always indicate a regression.

During the Regression phase, performance engineers do the following:

  1. Update the performance environment with the new drop.
  2. Update the previous load scripts to work with the new drop (if needed).
  3. Perform load testing, according to previous drop load configuration (data and flow).
  4. Compare results with the previous drop baseline.
  5. Publish results, focusing on negative trends and KPI breaks.
  6. Open performance defects (when regressions are found).

The Regression phase can be executed in parallel with the Pre-drop phase.

 

Update phase

During the Update phase, update the testing environment with the new functionality from the most recently deployed drop, as well as any new data and flow estimates generated during the Pre-drop phase.

When the new testing environment is up-to-date, it’s ready for a new performance testing session.

 

New baseline phase

The New baseline phase is the last phase in the Agile performance testing cycle, and ensures that the system is performance tuned for the following three months.

The performance engineer tests the system deployed with new drop, and configured for the test environment during the Update phase. These tests include the following steps:

  1. Running the performance tests.
  2. Publishing test results, focusing on KPI breaks.
  3. Opening performance defects for any KPI breaks.
  4. Saving the new baseline for the next drop.

Summary

Performance testing in an Agile environment, especially while implementing continuous delivery (CD) technique is different than the traditional performance testing lifecycle. In this blog I described a new Agile performance testing lifecycle which provides on time reports about the application’s behavior using the expected load for the upcoming three months. In the next post, I will describe how to analyze performance test results, and identify when to fix found issues.

 

Posted by:

Ben-Yehuda, Itay

Senior Developer

HP Software Agile Project Managment R&D

Comments
shay-tsadok | ‎10-14-2013 04:53 AM

Great article! I believe that this is the way to perform traditional PCOE in Saas world, in places were the organization has a budget to maintain production-like PCOE environment. 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation