Looking to enhance your execution reports? Have you tried automation reporting lately?

report2.pngLast night I was sitting around consuming pretzels (see my previous post for an explanation) and thinking about HP Discover Frankfurt.  I started doing something that is very dangerous for me—I started thinking. My thought: with Quick Test Professional (QTP) and now Unified Functional Testing (UFT) 11.5, why do we find it necessary to create external run reports for automation executions?

 

Over the last several years, I’ve seen a trend that has had a significant impact on automation performance—the compulsion of reporting outside of QTP and Application Lifecycle Management (ALM) or Quality Center (QC).  The level of detail ranges anywhere from a pass or fail text document all the way to the extreme of dynamic HTML pages.

I don’t believe this trend is occurring because of poor execution reports or the lack of built-in functionality of the automated tools for reporting. I think it has more to do with the consumer (or audiences of the report) and their lack of experience with reading these reports. This also has a lot to do with an automator’s experience with older versions of the results viewer. In the past, both issues could be addressed by two to three hours of training, and in some extreme cases, purchasing extra licenses.

 

When external reporting is necessary

 

There are several other circumstances that we use to justify external reporting:

  • As automation developers, it’s often hard to say “no” when it comes to designing new functionality
  • Sometimes a simple statement about customizing reports makes consultants see dollar signs flashing before their eyes
  • thCAS1FPGO.jpgMy kryptonite is both of these items, with the idea of building the ultimate reporting engine and someday marketing it under the wh4tsup_DOCument logo.  (has a picture of bunny in it)

 

In older versions of QC, it was required for a QTP results viewer to be installed on the client computer to view the detailed execution report. When dealing with business process testing and components, it was almost an inevitability for everybody to have access to the QTP Report. I could hash out several remedies that could help in automated execution reports. However with the release of UFT 11.5, I’d like to stay focused on its enhanced reporting capabilities for the time being.

 

If you haven’t looked at the latest reporting functionality of QTP 11.00 or the new enhanced reporting capability in UFT 11.5 in conjunction with ALM 11.X, I strongly recommend taking a look at the new reporting capability. I think you will be impressed with the level of detail in the results:

  • report5.pngThe screen recorder, allows you to view an entire video of your run, either frame-by-frame or as a complete movie. If a picture is worth a thousand words when dealing with defects, imagine what a developer could do with information collected in a video, which also includes the actions of the tool. The screen recorder/player is a free add-on which is included in the ALM install package.
  •  Monitor.pngThe system monitor report, gives you in-depth details of how the host machine behaved during execution of the automated script. This ability allows you to monitor the application under test behavior on the local host machine, which would allow the tester or developer to troubleshoot a myriad local performance issues, ranging from memory leaks to port integration with other applications.
  • The Data capture report, displays a still image of your application for the highlighted step. It provides a bitmap checkpoint image, a file content checkpoint comparison or other data. For example, if the step was performed for a Service Test or API Test step.
  • The Data report, contains the runtime version of the data table associated with your test or your ALM configuration. It displays the values used to run a test or configuration that contains data table parameters, as well as any output values retrieved from a test.
  • report3.pngThe Log Tracking report, displays a complete list of log messages that UFT received from your application during the run session, which can also be included when reporting a new defect.

 

Unlike custom reports generated outside of UFT or QTP, the reports mentioned above have no or little effect on the performance of your automated script. I also want to mention that you can choose not to use any reports from the reports viewer when creating a defect or simply generating external reports.

 

As much as I would like to create the ultimate execution report generator, it is sometimes smarter and simpler to use the built-in “reporter events” within the automation tool. It goes back to the human desire to build the ultimate mouse trap. But in all honesty, the only one that reads our heavily customized reports (with the cool logos) is us—the developer of the report. The level of detail, video, snapshots and cool graphics that our clients are looking for is already built into the results viewer.

 

Your chance to build the ultimate generator

 

If you could build the ultimate reports generator, what key performance indicators would you include when it comes to executing scripts. Barring the kitchen sink, I would challenge you to design or explain the next generation of reports. If you currently use the built-in reporting capabilities of QTP and would like to share your reporting data collection functionality, please feel free to tell me about it in the comments section of this article. If you have yet to find the reporting capabilities of QTP adequate, I want to hear about your resolution.

 

Come meet me at HP Discover

Also, I will be attending HP Discover in Frankfurt from December 4-6. I would love to meet with you and discuss your ideas.  You can find me eating pickled herring or schwarwurst in the experts lounge. (Remember to carry mints with you or you will suffer)

 

footer-discover2012-sw-600x60.gif

 

Meet the Experts Day 1

Tuesday, Dec 4, 2012

3:30 p.m.-4:30 p.m. CET

HP User Community Lounge

Can you go? Join our event on Facebook!

 

Meet the Experts Day 2

Wednesday, Dec 5, 2012

11:30 a.m.-12:30 p.m. CET

HP User Community Lounge

Can you go? Join our event on Facebook!

Comments
David Hartley(anon) | ‎11-30-2012 07:53 AM

I really like the video feature implemented in the reports, since using it it's made execution analysis much easier. I have to say that I haven't needed to use any of the other additional reporting features implement so couldn't comment on them.

There is one major flaw with the HP/QC reporting and that is around multiple iterations. If I've created a data driven test that runs 1000 iterations, the test is only represented once in QC. The problem with that is that it only takes 1 failure to mark the entire test run as a fail. What QC needs to do is break the iterations down and report on each one in test lab because each iteration is representing a different test case.

The only way I've been able to get around it is to design reports outside of QC where I can track iterations, but it is a painful mess of excel, VBA and OTA. If this feature does exist I would love to know, if not do you think HP could consider it?

Michael-Deady | ‎11-30-2012 08:18 AM

great question, this is one issue that has bugged me for so many years, and I have completely forgotten to test it in the new UFT 11.5. I'm currently getting ready to head to Germany but believe me; I will be unit testing that issue right away once I hit the Ground, and I will also be talking to R&D while in there.

QTP 11.00 and lower

Over the years like you, I've used a lot of workarounds, one of my favorites that seems to works without building external report instead of using checkpoints which only half past fail criteria; I've used passive checks like win_exist or check the run state of an object. Unfortunately, this doesn't bypass of those times when a GUI object can't be found during the set event, but it can stop those annoying failures on one iteration.

Quickest workaround for large data iterations for me has been a larger lab, breaking the data to run over several PCs or VM's so that it narrows the failure to a subset of the data.

I've contact as a friend who told me that he figured out a way to intercept the failure before it was written to the report, which sounds great, unless he's altered the function libraries of QTP?

To this day, most people still manually change the pass or fail within quality center. I will definitely get back to you no later than Friday of next week with an answer to your original question as it pertains to UFT 11.5.

I know a lot of the viewers of these blogs are both R&D and marketing, which drive updates and patches for both new and older versions of the application.

 

I hoping that R&D has some Feedback for you.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Michael Deady is a Pr. Consultant & Solution Architect for HP Professional Service and HP's ALM Evangelist for IT Experts Community. He spec...
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.