Part 1: The Framework that binds us
I’m baaack! Well, this is my first blog that I have written in a long time. And what better way to make friends and influence people than to come out with a broad generalized statement. I am sure this statement is sure to work up some of the readers out there. Before you grab your pitchforks, let me start with my background and then get into the subject, this way you can get to know me and what I am about.
I have been working in the Quality Assurance community for quite a while. And before I started as a legacy Mercury consultant and then as an HP consultant, I was a customer like you. So rest assured, I have sat on your side of the table and experienced when someone from HP came in and said “you should try this” or “look at this new-fangled gizmo application that will solve EVERYTHING”. Trust me; I have rolled my eyes at those statements too.
My years in the trenches working to make my job smarter and get the maximum reuse out of every little line of testing code that was created, has given me some good insight onto what has been done and what is going on with today’s applications. As a consultant, I have seen all kinds of customers: from brand new with a single tester, to very mature COE teams, to the large scale firms that bring Testing and Automation-as-a-Service into organizations. I understand your pains and gains with those different organizations. And what I have seen is very interesting—we are making this automation thing way too hard on ourselves.
Why are we making automation so hard?
So what does that mean exactly? There is a need for reusability in any form of application development today, and test automation is no exception. We want to rapidly turn-around testing to keep pace with the rapid change, independent change, Agile, extreme programing or whatever your current flavor of development is today. We have teams, large and small, that need to be trained and mentored in our practices to get the most out of the investment we make. I have seen from organizations that there are many different reactions to these needs: to provide more coverage, faster turn-around and leveraging of our talent pool to keep this total cost of test ownership down. All of these are needed in today’s business.
HOWEVER, I believe the solutions being provided and developed often re-invent the wheel—or in some cases the square wheel—and the initial problem gets lost on the new focus on the solution. Let’s talk about where we see these issues in the field over the next couple of weeks and months.
Going after the big fish in the pond
So let’s go after the big one first, the dreaded ‘All-In-One Frameworks’. It seems everyones has a framework and they love to talk about. They also love to talk about the need for a framework in automation and how automation frameworks allow them to turn around testing in such a short time and so on. I can honestly tell you that this is the largest issue in test automation complexity that I see today in the companies that I work with.
Hands down they are responsible for the bloat of test scripts and performance issues, not so much in the time to create test, but the time it takes to execute testing and maintain testing. A framework danger lies in that they start off as good idea, simply because we want standardization of our development from all teams and all members. We want to do more with less, we want to standardize development and we want to leverage expert resources across our junior resources and we see a framework as the solution.
The issue is that they grow way larger than they need to be, and they end up becoming the focal point of our testing, not the testing itself. Then the usefulness of the framework is bogged down by the complexity of the solution. All the cost savings and standardizations that we gain become lost in the quagmire of maintaining a framework. Every benefit that we saw on day one of our framework inception is lost rapidly, as we make that framework a one-stop shop for all challenges, and lose focus on the testing solutions we want to provide.
As an example, we want to create a framework that has reusable functions for interaction with standardized objects. This starts off with the best intentions and works great. Our tests have a way to interface with a set of dynamic or custom objects that the current toolset does not allow non-expert users of the tools to work with. And now all users can leverage these functional libraries, so we have standardization in tools. Then we want to make them even more flexible to work with more objects, because we had such success with the first go round.
We continue down this path, over and over on many different versions of the product that we are testing, replacing more and more of the act of ‘scripting’ with the ‘framework’. In the end, the ‘framework’ ends up re-creating the functionality you would see in the core Quick Test Pro (QTP) or Unifed Functional Testing (UFT) Object Repository and HP Business Process Testing. The framework offers none of the operational benefits and all of the performance issues.
The structure of the framework did not allow for innovations from the product set, or allow users to see the multiple solutions available to address testing goals. Instead it forced them into a set of choices that are code intensive and not optimized for the current generation of the tool. The framework concept ends up making the act of creating a test really not much more than entering lines of objects and actions into an excel spreadsheet or pulling in 100’s of test modules without really understanding what that test is really doing. Our testers lose the ability to really understand what a test is doing (and lose the ability to troubleshoot that test effectively when a defect occurs) and we no longer leverage the tool’s functionality to optimize performance over versions of the product.
Other options to consider
I would rather see a set of guidelines, not a framework for development:
- Guidelines allow for flexibility of solutions to leverage what is out there, without forcing us down a path of using the same solution for every issue.
- Guidelines allow us to have smarter testers: they need to make decisions, understand the problems in front of them, and we get the value of having different solutions and ideas brought to our testing.
- Guidelines understand what needs to be standardized, and when to review them as each newer version of your AUT comes out, as well as new testing products come out.
The removal of those ‘all in one frameworks’ from each test will streamline execution and automation by using just what you need, and not having more than you want in each test. Testers become better by focusing on what is needed for each solution, when to reuse existing tool solutions and when to promote something for reusability. It gives them the option to be creative and not blindly follow a ridged structure of testing—which does not promote innovation within our quality teams. This is what we need, a simpler and cleaner approach. An approach that lets the automation tools do the jobs that they are optimized for and allows testers be the thought leaders in those solutions. It maximizes our investment in the tools and our people, which is a win-win in anyone’s book.
So there you go, I hope you found this entertaining, somewhat informative, and/or at least though provoking. I hope this will spawn some conversation and some ideas, or at least give you a quick read between meetings. Next up, I want to look at modularization in testing and how it can be both under- and over-utilized in today’s automation.
Let me know what you think about the “framework” vs. “guidelines” debate in the comments section below. Thanks for reading and like Sergeant Phil used to say ‘Hey, let's be careful out there.’