Exchange pricing system failure: Was it people, process, technology, or a partner?

imagesCAFXW8M1.jpgOver the past few days, I have been listening to the debate taking place about the so-called Nasdaq "Flash Freeze" on CNBC. For those who aren’t following this story, a problem in the exchange pricing system forced managers to take the whole trading system down. Today, the discussion centered around who is responsible, and why it took so long to detect and fix the problem. And this is not just an internal exercise—we now have exchanges blaming one another for the problem.

 

Hopefully, we will not get another Ford vs. Firestone fiasco; the only way to solve the problem is to conduct an objective review of people, processes and technology involved. At this point, managers should focus their attention internally rather than in asserting blame externally—otherwise, a better system cannot be built. If we want to truly improve the system, the answer is improving people, processes and technology.

 

People

Did people follow the process for detecting performance degradation or for fixing the performance issue once it was detected? How soon did people address the problem once it was detected? Why did it take 30 minutes detect the issue? Were key managers immediately alerted to the problem?

 

Processes

Were the right processes in place? Did the processes include a response aimed at solving the problem? What is the process for detecting a process issue? Was there a backup set of procedures for getting pricing information? Were people trained effectively on these processes? What types of root cause analysis for processes took place? What was the restoration/continuity plan? Should it be improved for future issues?

 

Technology

How quickly could the technology detect a performance issue? Was the performance issue automatically detected and an incident created as a result of the event? Why was old pricing data automatically stuffed into the system? Was there an automated run book to fix things once the problem was determined? Could other exchanges have acted as a backup for the pricing system? Other exchanges did not fail. What types of backup systems exist or should exist?

I liken this situation to something I witnessed several years ago. A major New York firm was providing poor performance to its customers during the holiday season—incidents more than doubled between October and November. And poor performance continued for months into the future. When we looked into the situation, we found that the firm had crammed in changes to avoid a change window. Was this issue people, processes or technology? It could have been all of them. People decided to rush changes, knowing that issues might arise from their haste. For the process, the change window could have been longer. Technology-wise, better and more automated testing could have been implemented. Only by asking these questions could this organization have improved.

 

What is my message to all involved?

Catch your breath. Stop blaming. Evaluate your people, processes and technology—then make the necessary changes. Finally, actively measure improvement via metrics such as the mean time between failures. If this improves and you measure leading-edge metrics, you can improve your system and its controls.

 

Related stories: E-discovery in the cloud: Can you find your data?

Global system integrator uses HP Executive Scorecard to improve customers’ business processes

Solution Pages: People and Process Management

Solution Pages: Technology and System Management

Solution Pages: Management Tracking and Alerting

Twitter: @MylesSuer

Labels: IT Performance
Comments
HeatherMackey | ‎09-06-2013 02:33 PM

Great post, Myles!

MylesS | ‎09-06-2013 03:21 PM

Thank you!

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Mr. Suer is a senior manager for IT Performance Management. Prior to this role, Mr. Suer headed IT Performance Management Analytics Product ...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation