Confessions of a VuGen “Scriptaholic”: Hard-won VuGen scripting Best Practices for APM

[#1]

Greetings all,

I’m especially pleased to share the “guest blog” podium today with the following great advice from my good friend and longtime APM/BAC expert Steve Katz (link).  Steve has been leading application test, quality and performance management efforts at Seagate for years now and in fact recently shared some of his best lessons learned about using HP Business Service Management to “master the cloud” on a recent webinar (link) with EMA analyst Dennis Drogseth (link).

If you missed the webinar (link) live, you owe it to yourself to find a little over an hour to get a better idea of how you’re going to have to evolve your mindset to manage services that cloud will increasingly play a role in….

Anyway, back to what’s at hand, Steve Katz’s great guest blog on scripting for APM best practices, with a special emphasis on considerations that folks experienced with using VuGen for load testing need to keep in mind.  I know you’ll enjoy it – last note, if you would like to guest blog just let me or anyone else on the HP APM team know and we’ll connect – of course we also always welcome you to become part of all of the discussions happening here…..

Cheers,

Colin

Colin Fletcher
Sr. Product Marketing Manager, Application Performance Management/BAC
Hewlett-Packard Company

www.twitter.com/ColinF_at_HP
www.hp.com/go/apmblog
www.hp.com/go/apm
www.linkedin.com/in/colinfletcher

Confessions of a VuGen “Scriptaholic”:  Hard-won VuGen scripting Best Practices for APM/BAC

By Steve Katz, IT Manager, Seagate

Recognizing differences is an important trait in life; teaching your children right from wrong, noticing when you and the bank don’t agree on your checking account balance, and playing the “spot difference puzzle” in the Sunday comics (I always miss the hats and the spots on the dogs).  This skill also comes in very handy when writing code and is especially usefully in VuGen scripting.  Anyone who has mastered manual correlations in their VuGen skills will appreciate this.  But the differences are not only evident how you write your scripts but also why you write your scripts.  The “how” and the “why” differ between loadtest and Application Performance Management (APM) scripts.

When making the transition from loadtest scripts to APM scripts there’s a few minor differences that drive you to your goal of an effective script.  Loadtests are designed to stress the system and leech out bottlenecks and bad code.  To prepare for load tests you may write scripts designed to simulate the process of hiring/firing employees, buying or selling product, moving materials around in your supply chain, accepting stock options, or ordering office supplies.  In short, we can change the state of the world as we know it because we are operating in the realm of our test environment. 

In APM, our “production life,” we do not have the luxury of “changing the outcome” of our critical business processes. We are monitoring in live, production systems so we are often relegated to the “look but don’t touch” paradigm.  We can poke, prod, and manipulate but need to stop short of actually hiring that new employee or purchasing one of our company’s wonderful products.  The hard part is writing scripts that are reliable, instrumented at the appropriate level, robust enough to handle some variation, and most importantly, get to the heart of the critical business process at hand.

For our team recognizing the subtle differences between loadtest and APM scripting was not something we figured out over night.  It was harder than the Sunday comics puzzle but far easier than that “teaching your kids right-from-wrong” thing.  Here are a few epiphanies we discovered along the way that may be helpful;

Size Matters: Customers will come to you with APM monitor ideas that look suspiciously like their User Acceptance Test plans.  An APM script is not meant to be an exhaustive validation of the application’s functionality.  Rather, it should include the barest essentials to validate the ability of your IT infrastructure to deliver a business service.  Think Small!  Even in our most complex applications we try to limit our transaction count to 10-12 items – this can sometimes have licensing benefits too, if you think about it. 

The Emperor’s New Clothes: The rule of thumb here is to beware of both naked Emperors and naked transactions.  We all know what a naked Emperor looks like (sorry about that visual), but may not be able to spot a naked transaction as easily.  An important step or transaction in a script that for whatever reason, is not encapsulated in a named transaction, is considered a “naked transaction”.  In APM (without some clever error management) the gold standard is the named transaction.  An application failure in a step not encapsulated within a named transaction will generally yield a silent failure, or script failure that is not detected.  You can test this yourself.  Write a simple script (without named transactions) against a web application that requires a login.  Use the wrong password and see how your script behaves in VuGen. It will likely succeed despite the injected error. 

Also, the rule of thumb in transactions is generally one transaction per web call. Cramming multiple application/business process steps into a transaction will make root-cause investigation difficult and will bias performance measurements in the wrong direction.

Test the Waters: As a corollary to the Emperor’s New Clothes, inside every named transaction there should be at least one content check on the page or item being downloaded.  In most cases a web_reg_find transaction will suffice.  This ensures that you get what you expect for a particular business process step.  A transaction without a test is another area with a potential for a silent failure. 

In some cases you may need to look at the content or size of a downloaded component.  Remember, you can also test for the absence of content (i.e you don’t want to see the existence of some specific content – like a coded error message) using a flag in the web-reg-Find call.

You Better Think!:  While this directive works well in the old Aretha Franklin song (of the same name), it doesn’t work  so well in our scripts.  Generally the lr_think_time function is used in performance testing to help distribute user load, but is generally not used in application performance management scripts.  First, we really don’t need to worry about loading an application with a single user running through a business process with no think times (if you need to worry about that – the application being monitored likely has a litany of bigger problems).  Second, we want to ensure that the timings reported in APM via our scripts are an accurate account of the user transactions and not biased by any embedded think times.  If you absolutely, positively must include a think time or two, make sure you place them between rather than within any transaction.

Stop, Do Not Pass Go - This is not exactly a script guideline but it is certainly a script execution best practice.  When configuring your runtime settings in VuGen take careful note of the “Continue on Error” switch in Runtime Settings>Error Handling.  Generally, this is best left unchecked (which is off), as in most business processes when you fail, you are done with that attempt.  For instance if you fail login, it makes no sense to press on and attempt the next transaction.  It will fail and mask the location of the real problem if you continue on error. 

The only place continue on error IS appropriate, is the script that is looking at unassociated web pages that are not part of a sequential business process.  We monitor over 50 applications at our site, and the only script that has continue on error enabled, is a script that monitors our internet POPs by navigation to three popular (unrelated) public sites.

So there you have it (and I hope it was useful).  Since APM scripts are really designed to do something very different from their brethren loadtest scripts, the objectives and scripting techniques are generally very different.  This is not to say that we cannot leverage our load test scripts for use in APM, as we often can.  Most of the underlying assumptions hold, however you will need to remove variable parameterizations, think times, and script action-flow to make a loadtest script APM-worthy.  But, even if you can’t directly leverage the loadtest script for APM use, the experience gained in writing the loadtest scripts, knowing what correlations are required, and understanding the idiosyncrasies of the test application’s code are (as they say in the credit card commercials) priceless…

Knowing the differences between the two test disciplines will make you a better test engineer and script writer.  It will help you focus on the essence of what’s important in both load testing and APM scripting, and will ultimately make you better at both.  Yes, recognizing differences is a great thing.  I’m going back to my Sunday puzzle; I’ve found three differences… just four more to go.

Best, Steve