Load testing and continuous integration for Agile and non-agile environments

(This post was written by Gal Shadeck and Yuriy Pavlovsky from the LoadRunner R&D team)

 

As more and more software companies and departments switch to Continuous Integration and Continuous Delivery practices, it becomes crucially important to integrate load and performance tests into the process. Developers and DevOps engineers must make sure that each new build which goes directly to QA, or even directly to the end user, doesn’t introduce performance regressions. Additionally, it’s sometimes important to include existing unit tests in load-testing scenarios, or perform additional analysis on unit test execution.

 

That’s why LoadRunner 11.52 contains a set of new features aimed at continuous execution of performance tests and integration with unit testing frameworks.

 

Jenkins Plug-in

 

The HP Application Automation Tools plugin for the Jenkins continuous integration server has been available since the release of UFT 11.50. Its second version, released to coincide with LoadRunner 11.52, provides a simple mechanism for executing LoadRunner Controller scenarios as part of your build script.

 

Anyone who has experience with LoadRunner scenarios knows that each run produces a large amount of data, which gives the full picture of the AUT’s performance and is best understood using LoadRunner’s Analysis application. However, when executing a load testing scenario as part of Continuous Integration, Deployment or Delivery, the most important thing is to get a clear “pass/fail” indication of whether the application’s performance has been hit as a result of recent changes in the code, without doing any data processing. That’s why only scenarios which have built-in service-level agreements (SLA’s) can be run via Jenkins.

 

Here’s how this can be achieved (we assume that if you’ve read this far, you know what Jenkins is and are familiar with its basic features):

 

  • Install the plugin by going to ‘Manage Jenkins’, then ‘Manage plugins’.  In the ‘Available’ tab, look for ‘HP Application Automation Tools’, and install it:

P1.png

  • Select an existing build project or create a new one. Typically, functional and performance tests should be placed within the product’s build script, following the build and deployment sections.
  • Add an “Execute HP tests from file system” step:

P2.png

(note that the “Execute HP tests from HP ALM” step supports executing functional tests, not load testing scenarios; we’re considering adding this feature to a later version of the plugin)

  • Set the step properties:

P3.png

 

  1. Tests – list of paths to scenario folders or .lrs files, separated by new lines. Yes, more than one scenario can be executed in a single build! As mentioned above, only scenarios with defined SLA’s are allowed (others will be marked as “failed” without execution). Also note that the paths you supply must be absolute and reachable from different computers within your local network, as Jenkins can use slave machines for test execution.
  2. Timeout (in minutes) – timeout for the entire build step. If the scenarios included within the step are not completed after the specified number of minutes, the entire step is marked as “failed”.
  3. Controller polling interval (in seconds) – polling interval for checking the scenario execution status. This will affect the frequency of status updates in the Jenkins console during the run.
  4. Scenario execution timeout (in minutes) – timeout for lengthy parts of a scenario execution, which include running Vusers, collating results, analyzing results and shutting down the Controller. If any of these operations within any of the selected scenarios is not completed after the specified number of minutes, the step is terminated and declared “failed”.
  5. Ignored error strings – scenario execution is considered failed either if the SLA is violated or if an error occurs (in addition to the trivial cases, such as scenario not being found at the specified location). In order not to fail the scenario on insignificant errors, a list of error messages, which can be safely ignored, can be supplied. If an error occurs, and at least one of the strings in this list turns out to be a substring of the error message, the error is disregarded. For example, if we add the word “CPU” to the list, the often encountered “80% CPU use” error will be skipped. No need to use asterisks or any other special characters.
  • Define publishing of the test results as a post-build action:

P4.png  

 

P5.png

 

  • Once the Jenkins project is ready, run it, either immediately or using the scheduler
  • During the execution, you’ll be able to view its progress using the Jenkins output console. The moment the job has finished, you’ll get an immediate indication of the tests’ status:

P6.png

 

P7.png

 

(note that the zip file that appears under “Build Artifacts” contains the full run results and can be processed using Analysis)

 

 

IDE DevOps plugin

 

To support deep and effective integration of unit or functional tests with LoadRunner infrastructure we provide addins for Microsoft Visual Studio 2010 and Eclipse with LoadRunner 11.52. You can install these from the LoadRunner 11.52 Additional Components location.

  

Let’s suppose that you have some existing code with business logic and unit tests for it. The following example is based on the Microsoft Visual Studio 2010 IDE with the NUnit framework. For Java-based frameworks with the Eclipse IDE, the Continuous Delivery plugin provides exactly the same functionality.

 

P8.png.jpg

 

Now, extending your test with all the power of LoadRunner’s infrastructure is as easy as clicking on “Add LoadRunner API reference” in the “DevOps Vuser” menu, which is created when you install the addin. After this simple action, you can use LoadRunner functionality such as transactions, messaging, think time, etc. in your code. 

 

P9.png

 

Alternatively, you can use the context menu of your unit test project in the solution:

 

P10.png

 

In the code, instantiate the LoadRunner API:

 

P11.png

 

.. and now you can have LoadRunner transactions in your unit test code!

 

P12.png

 

It is important to note that this change doesn’t break the way the test works.  As long as the referenced libraries are available, it will work the same way it used to work before.  If you execute the test as usual, the additional LoadRunner code won’t actually do anything.  But now you can run it as a LoadRunner VUser!

 

P13.png

 

The result of the running this test module as LoadRunner script is redirected to the “LoadRunner Information” output window in the IDE:

 

P14.png

 

 

Unit test support in the Controller

 

To support execution of unit tests as part of LoadRunner load test scenarios, the LoadRunner Controller was extended with the ability to open not only LR scripts, but also system or unit tests:

 

P15.png

 

 After the test module is opened, a temporary script is created and shown:

 

P16.png.jpg

 

You can continue with scenario creation just as you did before with regular LoadRunner scripts.

 

 

Putting it all together

 

The features described in this article can be combined together. Take your existing unit test (or write a new one), extend it with the LoadRunner infrastructure, create a Controller scenario based on it and execute it as part of your build job.  In a few simple steps, you’ve protected your application from performance regressions.

 

(Note: if .NET is your technology of choice, make sure that NUnit and Jenkins are installed under the same user account)

 

 

Click here to learn more about HP LoadRunner

Click here to learn more about HP Performance Center of Excellence

 

WhatIsNewButton-11.52.png

 

  

Let us know what you think by leaving us a comment in the box below.

 

 

Thanks to Gal and Yuriy for providing this post!

 

Comments
venubasani | ‎11-15-2013 01:23 AM

Hi,

 

Action.c(6): Error: Failed to find SapGui component by ID "sbar"

 

Got error when I run oad test for SAP GUI using multiple Vusers .

Please help me to resolve this issue.

 

 

Thanks & Regards,

Venu

| ‎12-04-2013 12:10 AM

Hi venubasan,

 

I recommend that you open a support ticket for this issue, or post the question on the LoadRunner Support Forum.

Richa_S | ‎06-17-2014 04:11 AM

Hi Malcolm,

    You mention that - “Execute HP tests from HP ALM” step supports executing functional tests, not load testing scenarios - and that this would be provided in a later version. Could you please let me know when the load test scenarois would be supported?

    We need to automate the execution performance tests on HP PC. We are working on PC 11, so the "Execute HP tests using HP Performance Center" also does not work. In the plugin link, it mentions HP PC 12 as the version that can be used:
https://wiki.jenkins-ci.org/display/JENKINS/HP+Application+Automation+Tools

    Alternatively are there any command line options avaible to run the performance tests through HP Performance Center 11?

 

Appreciate the help.

 

-RS.

 

‎06-18-2014 05:20 AM - edited ‎06-19-2014 05:46 AM

Hi RS,

 

The current version of the plugin supports load testing scenarios.  See here for the list of supported tools and versions, and here for instructions on running LR scenarios from the file system.

 

I'm not aware of plans to support PC 11, but you can post on the PC support forums and ask about this and your query about command line options.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Malcolm is a functional architect, focusing on best practices and methodologies across the software development lifecycle.
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.