Automation Testing: Why are we making it so complex?

BSP_Pict.jpgPart 1: The Framework that binds us

 

I’m baaack! Well, this is my first blog that I have written in a long time.  And what better way to make friends and influence people than to come out with a broad generalized statement. I am sure this statement is sure to work up some of the readers out there.   Before you grab your pitchforks, let  me start with my background and then get into the subject, this way you can get to know me and what I am about.

 

I have been working in the Quality Assurance community for quite a while. And before I started as a legacy Mercury consultant and then as an HP consultant, I was a customer like you.  So rest assured, I have sat on your side of the table and experienced when someone from HP came in and said “you should try this” or “look at this new-fangled gizmo application that will solve EVERYTHING”. Trust me;  I have rolled my eyes at those statements too. 

 

My years in the trenches working to make my job smarter and get the maximum reuse out of every little line of testing code that was created, has given me some good insight onto what has been done and what is going on with today’s applications. As a consultant, I have seen all kinds of customers: from brand new with a single tester, to very mature COE teams, to the large scale firms that bring Testing and Automation-as-a-Service into organizations. I understand your pains and gains with those different organizations.  And what I have seen is very interesting—we are making this automation thing way too hard on ourselves.

 

Why are we making automation so hard?

 

So what does that mean exactly?  There is a need for reusability in any form of application development today, and test automation is no exception. We want to rapidly turn-around testing to keep pace with the rapid change, independent change, Agile, extreme programing or whatever your current flavor of development is today. We have teams, large and small, that need to be trained and mentored in our practices to get the most out of the investment we make.  I have seen from organizations that there are many different reactions to these needs: to provide more coverage, faster turn-around and leveraging of our talent pool to keep this total cost of test ownership down. All of these are needed in today’s business. 

 

HOWEVER, I believe the solutions being provided and developed often re-invent the wheel—or in some cases the square wheel—and the initial problem gets lost on the new focus on the solution.  Let’s talk about where we see these issues in the field over the next couple of weeks and months.

 

Going after the big fish in the pond

 

So let’s go after the big one first, the dreaded ‘All-In-One Frameworks’.  It seems everyones has a framework and they love to talk about.  They also love to talk about the need for a framework in automation and how automation frameworks allow them to turn around testing in such a short time and so on.  I can honestly tell you that this is  the largest issue in test automation complexity that I see today in the companies that I work with.  

 

Hands down they are responsible for the bloat of test scripts and performance issues, not so much in the time to create test, but the time it takes to execute testing and maintain testing.  A framework danger lies in that they start off as good idea, simply because we want standardization of our development from all teams and all members. We want to do more with less, we want to standardize development and we want to leverage expert resources across our junior resources and we see a framework as the solution.

 

The issue is that they grow way larger than they need to be, and they end up becoming the focal point of our testing, not the testing itself. Then the usefulness of the framework is bogged down by the complexity of the solution. All the cost savings and standardizations that we gain become lost in the quagmire of maintaining a framework.  Every benefit that we saw on day one of our framework inception is lost rapidly, as we make that framework a one-stop shop for all challenges, and lose focus on the testing solutions we want to provide.

 

Regaining focus

 

As an example, we want to create a framework that has reusable functions for interaction with standardized objects.  This starts off with the best intentions and works great. Our tests have a way to interface with a set of dynamic or custom objects that the current toolset does not allow non-expert users of the tools to work with.  And now all users can leverage these functional libraries, so we have standardization in tools. Then we want to make them even more flexible to work with more objects, because we had such success with the first go round. 

 

We continue down this path, over and over on many different versions of the product that we are testing, replacing more and more of the act of ‘scripting’ with the ‘framework’.  In the end, the ‘framework’ ends up re-creating the functionality you would see in the core Quick Test Pro (QTP) or Unifed Functional Testing (UFT) Object Repository and HP Business Process Testing.  The framework offers   none of the operational benefits and all of the performance issues. 

 

The structure of the framework did not allow for innovations from the product set, or allow users to see the multiple solutions available to address testing goals.  Instead it forced them into a set of choices that are code intensive and not optimized for the current generation of the tool. The framework concept ends up making the act of creating a test really not much more than entering lines of objects and actions into an excel spreadsheet or pulling in 100’s of test modules without really understanding what that test is really doing. Our testers lose the ability to really understand what a test is doing (and lose the ability to troubleshoot that test effectively when a defect occurs) and we no longer leverage the tool’s functionality to optimize performance over versions of the product.

 

Other options to consider

 

I would rather see a set of guidelines, not a framework for development:

  •  Guidelines allow for flexibility of solutions to leverage what is out there, without forcing us down a path of using the same solution for every issue. 
  • Guidelines allow us to have smarter testers: they need to make decisions, understand the problems in front of them, and we get the value of having different solutions and ideas brought to our testing. 
  • Guidelines understand what needs to be standardized, and when to review them as each newer version of your AUT comes out, as well as new testing products come out. 

 

The removal of those ‘all in one frameworks’ from each test will streamline execution and automation by using just what you need, and not having more than you want in each test. Testers become better by focusing on what is needed for each solution, when to reuse existing tool solutions and when to promote something for reusability. It gives them the option to be creative and not blindly follow a ridged structure of testing—which does not promote innovation within our quality teams. This is what we need, a simpler and cleaner approach. An approach that lets the automation tools do the jobs that they are optimized for and allows testers be the thought leaders in those solutions.  It maximizes our investment in the tools and our people, which is a win-win in anyone’s book.

 

So there you go, I hope you found this entertaining, somewhat informative, and/or at least though provoking.  I hope this will spawn some conversation and some ideas, or at least give you a quick read between meetings.  Next up, I want to look at modularization in testing and how it can be both under- and over-utilized in today’s automation.  

 

Let me know what you think about the “framework” vs. “guidelines” debate in the comments section below. Thanks for reading and like Sergeant Phil used to say ‘Hey, let's be careful out there.’

Comments
MichaelDocDeady | ‎02-13-2013 07:36 AM

Is the weekend mafia taking over HP IT expert community page? I Feel like I'm reading a script from the Sopranos," Sil, break it down for 'em. What two scripting methods traditionally been full-proof since time immemorial?"

 

In this case, I happen to agree with Mr. Brian “Tony” Palagyi there’s always a trade-off when it comes to automation when we try to make it simpler for the user or we attempt to make it as hearty as the application it test we sacrifice performance and easily maintainable automation framework.  


@wh4tsup_Doc

thCAHZFT5X.jpg

vaishnavi_alm | ‎02-13-2013 09:48 PM

Interesting read. Can't agree less with you. When it comes to the Guidelines vs Frameworks contention, guidelines would be the "Less is more" solution.

| ‎02-14-2013 05:30 AM

But isn't that what we are shooting for?  We want a simple solution that allows greater flexibility for all scripting and testing needs while not locking users into large legacy or unknown automation that doesn’t directly impact the current testing need.  With guidelines, you can regulate the creation and direction of automation and target development of reusable actions where there is greatest benefit, rather than forcing them into the framework where they are either used or not, but still account for the majority of code and performance bloat within automation.

vaishnavi_alm | ‎02-14-2013 08:23 AM
Absolutely. The combination of "Test automation + business process tests + test management" is a light-weight framework in itself, giving the tester the flexibility to design tests focussing on testing the AUT, rather than the overkill of maintaining the automation framework and many lines of scripted code. With a jazzy framework, the ROI gained in test automation is eventually eroded by the maintenance effort, in addition to the performance trade-off. Guidelines help maintain the standards and also give room for creativity.
MichaelDocDeady ‎02-14-2013 09:46 AM - edited ‎02-14-2013 09:47 AM

You may or may not  know from my blogs I a big proponent of business process testing and like you I believe BPT could be considered the foundation of keep it simple while embracing a structured framework. I don’t know if Mr. Palagyi is or isn’t it proponent business process testing; however I know I don’t believe in layers and layers of function libraries, static descriptive programming instead of a dynamic object repository, and Excel spreadsheets that are larger than most 19th-century novels.  And If Mr. P doesn't agree with me that BPT is the best  thing invented since the spill free coffee cup just know that he's wrong and I'm right

 

Thanks

@wh4tsup_doc

doc.jpg

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
I have more than 12 years in IT, with over 10 years working with the HP Quality Management suite of tools—seven as a Professional Services c...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation