Centralizing the test automation team

OK, so before I start babbling on about all the aspects of testing that I think are really interesting, and before you jump in and decide whether this blog is worth spending the time on, I thought I’d give a short intro about why I decided to start this blog and what I hope for it. Later on, starting with this post and probably continuing with later ones, I’ll write about a change that is happening, for a while now, in the quality assurance industry and testing in particular in mid to large IT shops that is changing the way testing is being done, managed, planned and funded – Centralization or a formation of a ‘Center of Excellence’ (CoE).

Why does this blog exist?

Well, during the years of my professional life I experienced endless conversations, debates, arguments, failures, successes, epiphanies and unanswered enigmas. All those shaped my view on what I professionally do and how I choose to do it. Although I read blogs, blogging myself was as far from what I was planning to do as me being able to remember where I put my cell phone. After many conversations with customers of mine as well as our solution-engineers that are working with them, I understood how much such a blog can help not only me, in getting my thoughts and understandings well defined and put through but also allow amazing knowledge to be shared across the huge customer base and beyond it as well as continuously learn from you - the readers, through your replies and not just wait for the next meeting or customer visit. Of course this can happen only if you take the time to read and reply. Any feedback is welcome. In this blog, I will focus on thoughts and problems related to both manual and automated functional testing (FT).


OK, enough wasting electronic web pages…let’s get to it.

Centralization of Automated Functional Testing

With this post, I’d like to start describing a change that has been happening in the industry around centralization of quality assurance activities and automated testing in particular. For few years now, IT shops have been gradually centralizing many of the QA related activities, starting with shared administration of QA tools such as Quality Management, Source Control, Defect Management, Requirement Management and others. This was followed by centralization of infrastructure and whatever made operational and business sense. The centralized group is often referred to as the ‘Center of Excellence’ or CoE. During this whole time many shops continued to have FT, manual or automatic, silos in their business units either due to the domain expertise that was needed for each business unit or due to the relatively small size (if relating to automation teams) and in some cases low exposure to upper management.


During the last 2-4 years, I am seeing more and more organizations making the automation team part of their CoE or centralizing it in one way or another. This means these companies have a single team (sometimes few teams each supporting few business lines) that owns the automated FT tool, responsible to manage licenses and administrate them as well as distribute the tool if needed around the organization. These are the very basic functions that can cut down on cost, license consumption and maintain consistency in deployment. However, where I see the most value is where these teams also own the best practices that are then shared, and implemented in the different testing teams and the actual centralized the development of automation skill sets. Having these capabilities in a specialized team, allows the organization to learn and grow its test automation initiatives faster. It also means that there needs to be a well defined process for allocating the shared resources and sharing the knowledge or output throughout the organization.


Setting these processes and embedding them into the organization is not an easy task and usually requires effective documentation, hours of guidance to the relevant people in the business units and continued support by the automation CoE to the different business units. This also means there needs to be a strong enough owners from the recipient side (the line of business) that can observe these best practices and scale them further. Absorbing and growing the knowledge is best done by what I call a ‘Local Champion’. This key person seems to be one of the most important people for success of an automation initiative within the line of business. Their technical capabilities, partnership with the automation CoE and the level of guidance and support the CoE gives is a make or break in the success of such an effort.


Due to the difficulty of this partnership I have seen models where the automation CoE decided not to hand over the ownership to the business unit but rather automate in full the test suite and hand it over ready to run. This is a valid approach in my opinion as I have seen organizations using it with great success but I have to say that it does has the down side of slow scaling and can position the automation CoE as a bottleneck. The advantage of this approach is of course the maximization of knowledge re-use. It also makes reusability of test assets (reusable libraries, functions, etc.) much easier as there is 1 single team that creates and manages everything. With the other approach of having the CoE hand the ownership at some point to the local champion, there needs to be an on-going contact to keep circulating the knowledge that is gathered in the different projects and a well defined process to document the reusable assets that are made available to everyone. Nevertheless, even if reusability is not perfect in this approach, when the local champions are strong, this has great success and scales up fast.


With all of the above in mind, the CoE needs to carefully choose the tools that allow to effectively creating, managing, sharing and standardizing the processes, best practices and test assets.


To recap shortly, the key points, in my opinion, to a successful automation CoE are:



  1. Centralized ownership of the FT tool

  2. Standardization of test development process and best practices and the ability to share those across the organization

  3. Strong Local Champions in the lines of businesses (if the ownership is passed to the project).

  4. Strong and on-going relationship, having the automation CoE support the local champion and facilitate knowledge sharing between the local champions.

  5. Well defined processes that allow sharing of reusable assets between tests in the same test suite and between projects when possible.

  6. Automated functional testing tools and test management tools that support the needs above.

That’s it for this time…


Roi

 

 

Comments
(anon) | ‎01-19-2009 07:53 AM

Thank you for a very interesting post. Looking forward to the next one.

(anon) | ‎01-19-2009 01:10 PM

In principle I agree with the 2nd approach as the most sensible, due to the fact that local project teams will normally come under pressure to retain a sense of autonomy and maintainability due to local issues.  Especially where geographical distance from the local team to the CoE is a problem and the CoE is seen as a bottle-neck to the local project by Project Management.

In my mind what must be upheld is that the CoE maintain the building blocks behind structure in terms of test assets, and principles of the test approach and roll-out.  Such that they maintain the philosophy and the strategy behind the ‘whole test’ frame-work.  

Local maintenance and roll-out is of real importance, but only a means to and end to help solve ‘local’ issues, which are normally project related (time-scales and resources).  Any standards or rules introduced by the CoE must never be broken without prior consent.  And then these changes should only be an exception to the rule; the CoE being seen as the source of ‘governance and ‘steering’ to the overall approach and idea of automation.

(anon) | ‎01-21-2009 07:55 AM

Good start to your blogging endeavors. I have actually experienced much of what you pointed out first hand. I was there when your company first introduced a testing center of excellence. I happen to be a part of the automation aspect of that CoE. We operated out of best practices, standards and guidelines and had a unified approach to our automation that enabled support of multiple lines of businesses. We have seen great success and experienced frustrations as well. But in all, we have proven to the whole CoE that automation is invaluable and we receive constant pressure to always improve and figure out ways to get better automation penetration. We support hundreds of applications so each line of business offers their own challenges. We work with MainFrame(Terminal Emulator environments), Web, .NET, Java, Flash, ActiveX, you name it.  Currently, we are also trying to introduce a keyword driven framework in addition to the concept of our automation resources automating a project and handing it over. We are actually trying to enable the lines of business to provided their own SME to automate their applications after our automation SMEs set up the infrastructure...kinda BPT without BPT but can work with BPT, without the fee (yes I meant to rhyme!!). Anyway, I'm definately curious as to what everyone else is doing out there and look forward to reading about this topic and hopefully I can add something here and there.

(anon) | ‎01-22-2009 09:39 PM

Roi, nice blog and I like the topic.  This is something that I think many organizations are struggling with.  

(anon) | ‎01-29-2009 10:05 PM

Excellent!  I love the fact you provide a recap/key points...very helpful :smileyhappy:  Keep it up and looking forward to your next posting.

(anon) | ‎02-06-2009 04:19 PM

Good to see a blog direct from QTP product manager.

It would be interesting to see your future posts. 've subscribed to RSS.

Esteemed Contributor | ‎02-11-2009 01:28 AM

Roi, one thing you haven't mentioned is the relationship between the Automation Team and the Test Teams. Aside from things like Data Load, the point of Automated Testing is testing. This is probably one of the things Automation teams do poorly. Teast Teams (usually) are good at testing because they understand the application, business requirements, the art of testing, etc.

To get a good automation engineer and give them business domain knowledge, or to get a good business domain tester and give them software development and application technology knowledge is not a trivial thing. It is not something you would do unless they will be testing the application for a long term - which goes counter to the idea of the CoE for anything more than process and standards.

Like LaBron, I prefer the approach of making the CoE a factory, delivering automation assets for use in test fabrication by SMEs - only I do use BPT in preference to home grown or less powerful frameworks.

Will Anderson | ‎02-12-2009 06:49 PM

Hi Roi - terrific blog! I work in a 100+ year old insurance company that has only had QA for 3 years. We use Quality Center, Performance Center/LoadRunnner, and Quick TestPro as our autoation basis.

I am leading an effort to make a CoE for test automation and performance testing that will compliment our FT group, as well as support our maintenance and patch management groups (both run by QA). I have good technical buy-in from development and IT infrastructure, and have acceptance of my approach to create standards, guidelines, templates, reports, and libraries as the core of the CoE. We most likely will not be integrated with RE as they are only a year old and have enough on thier own plate.

Though the business owns the applicaitons functionality via day to day use, and upgrade requests (and new product functionality), IT infrastructure owns the care and feeding of the business systems, and in very few cases has the expertise to test prior to and after deployment.

Do you have any words of advice concerning ROI selling points (especially to QA management and IT executive management) - we need to keep lights-on costs low, maintenance costs flat or reduced, and bring on new pieces of the enterprise architecture, but as you mention in your blog we need the help of others. Any input would be great!

msteffensen | ‎03-10-2009 12:23 AM

Roi - I'm confident in your knowledge of the QA space but then you know that!. My addition to this discussion is the topic of data management. At its core, tests needs to be separate from the data the will use. You can run the same test with three different sets of data and evaluate the results separately. In addition, you need to be able to manage positive and negative tests. A negative test runs with data meant to fail in a predictable way. Preferably one test can be used to perform both tasks but I'm curious how organizations handle this. Do they build two tests or use one test and report the predicted failures in a "passing" way?

(anon) | ‎05-11-2009 03:26 PM

Great blog.  I have worked as part of a centralized automation team for the last 9 years.  It has been a struggle to keep the team centralized.  The power above are trying to push the automation back to the development teams.  I am a firm believer that our current method is what is best for the organization,but I am struggling with providing any metrics that can prove my point.  Does anyone have any metrics that would show how centralizing automation is in the best interest of the company?  Thanks!

(anon) | ‎07-06-2009 12:51 PM

Excellent Roi!!!

Thanks a lot for your nice post!!

(anon) | ‎07-08-2009 09:07 PM

I want to say - thank you for this!

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation