Make sure you can measure the benefits of your quality management software

Recently, I shared my meeting with a CIO from a Global 50 company. When I discussed with him the importance of IT performance management, this CIO said that no one talks to him about IT performance management. He went onto say that having a performance management system would ensure he gets the business transformation and ROI out of the IT toolset purchases that he makes. “What I need is best practice KPIs to measure along the journey,” he said. Given this, I will turn my attention in this blog to quality management and what measures matter for this activity.

 

 

 

What objectives do customers have for purchasing quality  management software?

 

Once again, I’ve asked some of my savvy customers what their reasons are for purchasing a quality management tool.  Here is what they told me:

 

1)      They’re looking to decrease the man hours required for the business and IT to develop scripts for testing (functional, regression, etc)

2)      They want to decrease the man hours used for defect tracking and reporting. This includes traceability of quality from requirements, development management, and performance validation.

3)      They want the ability to ensure there are no performance issues with sociability testing for enterprise applications.  One of my contacts told me, “Right now, an army of people perform testing (functional, integrated, regression) from IT and the business. And the business is always complaining that they do not have resources for the additional product testing.”

4)      They’re looking out for the bottom line. Another person said, “These are the justifications to purchase a tool that frees up resources since the tools have a better ROI than bringing in high priced contractors at $200 per hour for testing.”

 

These are great objectives. And a balanced scorecard can demonstrate that IT organizations are achieving these benefits from a quality management tool. Clearly, such a scorecard should be based upon the business goals set for such a purchase. For purposes of this blog, we will use the four objectives above to guide us.

 

What would go into a Quality Scorecard?

 

So if we follow the four-quadrant model for Norton Kaplan, what key performance indicators (KPIs) should be in the scorecard for a quality management purchase?

 

Quality Scorecard.jpg

 

First Quadrant—IT Value

The first quadrant in the scorecard focuses on IT value.  Given the above objectives have a strong pecuniary bent to them, I recommend a series of KPIs here. These include the percent of projects at budget risk and the percent of change in project costs. Now, some of you may object to project measures here, but quality testing is a direct expense to the cost of a project. And since modern quality management happens as features are in fact developed, it is an appropriate measure of cost dimension of quality. The number of escaped defects is on the value side of the ledger. Now this is a strange name but it is about the number of defects that are discovered post release. This is the ultimate measure of value for quality because quality simply should capture defects prior to release—i.e. this number should be very small. The final one for value is % of Actual vs. Planned Tests Executed. This is a way of measuring whether I do what I say that I am going to do in terms of delivering value.

 

Second Quadrant—Customer

Moving to the customer quadrant, I want to see support for goal number 2. Given this, KPIs that make sense here look at the quality in responding to customer needs. Here I would look at two areas. The first includes the % of Tested Requirements and the Average Cycle Duration. The first KPI tells me the likelihood of a defect escaping into production and the second KPI tells me whether I am reducing the cycle time which is also relates to goal number three above. The second area is Defect Number of Escaped Defects and Average Time to Resolve Production. For customers, this should be “where the rubber meets the road”—I want limited escaped defects and when a production issues happen, the ability to quickly resolve it.

 

Third Quadrant—Operational Excellence

For the operational excellence quadrant, many KPIs make sense to track change. Specifically, we want to measure the quality of processes and the improvement made to them.  These include things such as: % of Reopened Defects, % of Tested Requirements, % of Critical Defects, % of Requirements Traced to Tests, Average Cycle Duration, % of Automated Tests, Average Time to Review Requirement, Detected vs. Closed Defects Ratio, % of Rejected Defects, and % of Completed Tests. There are a lot of KPIs above. Together they measure the effectiveness and efficiency of quality and for that matter, project management processes. Clearly, increasing coverage and automation are critical to reducing cycle time.  And reopens and coverage are critical to the quality of process.

 

Fourth Quadrant—Future Orientation

In future orientation, we want KPIs that among other things measure quality employee satisfaction from the business process change for future orientation. Clearly, if the training is not taking place, employee frustration and sentiment could go up and down respectively. These together show if an innovative, future oriented IT organization is being created.

With this said, we would like to hear back from you. What measures would you put on a quality management success scorecard? Where do you think we are missing something from the above list?

 

Related links:

 Feature:  Peak performance demands precision control

Solution page:  IT Performance Management

Twitter: @MylesSuer

Comments
Jonathan Smith 571(anon) | ‎01-03-2013 12:00 AM

Well, traking of requirements is a very important step for developing a quality software. Effective Traceability Systems provide facility of reducing errors in the software and helps in developing a quality product.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Mr. Suer is a senior manager for IT Performance Management. Prior to this role, Mr. Suer headed IT Performance Management Analytics Product ...
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.