Putting stabilizers in test automation and test execution?

[#1]

Part 4 – Testing Evolution Series:

I was recently asked, “How do you keep your automated testing scripts running in unattended mode?” Just so we are crystal clear on this subject, I don't. Most of my solutions are either built into the tools or they have been built by better people then me. I just leverage anything and everything I can to make my scripting resilient as well as dynamic.

I am not the first person to rely on the knowledge of others. I just read on Facebook that, “Thomas Edison invented the light bulb, but he didn’t invent electricity”.  In truth, over 23 other people are contributed with the inventing of incandescent lamps prior to Edison.         

If you use all or some of the Application Lifecycle Management (ALM) or Quality Center suite of tools, most of the functionality listed below is currently at your fingertips:

If you use all or some of the Application Lifecycle Management (ALM) or Quality Center suite of tools, most of the functionality listed below is currently at your fingertips:

1.  If you are manually testing as a team relying on each other to run a series of test, or you are using automation scripts, I would recommend using Execution Flow view with in ALM ‘s Test Lab module. Use the execution view to setup linked events or conditions, pass or fail criteria and with ALM workflow you can setup alerts and notifications. This will allow you to run automated or manual tests in a globally extended team or when you have scripts with automated dependencies.
2. Another tool inside ALM is the automation execution screen. This screen offers “On Failure” that allows you to set the number of times a test can be rerun before the system will fail the test set or move onto the next test. Within that same window, you will find the capability to call a cleanup script which will reset the environment before attempting to execute the script again. During automation testing, you can also setup email notification, ALM will automatically fire off an email if a test finishes with status "failed" or the environment failed.  These errors include: bad function calls, access violations, version incompatibility between application components, missing DLL or inadequate permissions. You can also have the ALM test lab send you or your team a detailed execution summary email from the same screen.
3. If your team is using Business Process Testing (BPT) it can also use flows in ALM to correct or redirect failed components. This can dynamically bypass problem areas with an application.
4. BPT, QuickTest Pro (QTP) and Unified Functional Testing (UFT) have the ability to activate specific recovery operations called “Recovery Scenarios” when these trigging events occur. The recovery scenario manager provides a wizard that guides you through the recovery scenario definition process.  These include unexpected events and the operations necessary to recover the run session. For example: you can instruct the tools to detect a popup message and click the OK button to close the message and continue the test. The recovery scenarios consist of three parts:
   1. Trigger Event – the event that interrupts your run session.
   2. Recovery Operations – The operations to perform to enable the tool to continue running the test after the trigger event.
   3. Post-Recovery Test Run Option – the instructions on how the tool should proceed after the recovery operations have been performed. Includes what point in the test the tool should continue, if at all.
5. The oldest known way to handle errors in Visual Basic Scripts is known as “On Error” statements. These allow the programmer to handle errors within a routine or an action. This works similarly to recovery scenarios; however, it has no fancy wizard tool or user interface. If you want to use the “on Error” routines you must know how to code in basic—which is similar to caveman drawing for most developers.  All kidding aside this sometimes has a downside like scripting overhead, poor use of memory and it is sometimes thought of as being archaic. In reality, if not done properly, this can cause a great number of errors in and of itself. Remember to avoid using words “on error go to” statement if you’re talking to C# or java programmers. They may laugh uncontrollably, which is considered harassment in my book, and you should report them to Human Resources.
6. My favorite way is to create and document my own error traps by using checkpoint or condition statements. Then call a subroutine or function which will work at the shell level of an application or operating system (OS) to clear the issue. When using this method you must be a little bit of a daredevil and you must know little more about the application and/or operating system then the average automator. I have great news; a lot of the people who like to do this type of work would also like you to know it; because we put a lot of this cool code on the web to show one another.

I talked about just six ways you can make your scripts more stable no matter the application or environment. Honestly, I could write a book on all the way’s you could script stabilization into your test. What I like to ask the readers of this editorial is to post your latest and greatest code, or solution. Think of it as a way to help the people who are struggling to sell their management on better ways to test or are stuck doing recorded and playback because they don’t have the time to do it right.          

Click Here to see part one of the unattened testing

what's up

Doc    

[#2]

Good article.  Are there any ways to set up a re-run based on certain error conditions?  For example if I have an error that occurs intermittently with error code Java Dump 404, re-run the script.  If I have an error that occurs like loan value was 4, expected 8, then the script should just fail and not be re-run.

[#3]

First of all, thanks, and yes there are several ways to check and rerun but my favorite is to use QC or ALM on failure function and in your cleanup script ID, the error to fix the problem. If you would like more information on the "on failure" functionality, just drop me email and will send you some more information.