Business Service Management (BAC/BSM/APM/NNM)
More than everything monitoring, BSM provides the means to determine how IT impacts the bottom line. Its purpose and main benefit is to ensure that IT Operations are able to reactively and proactively determine where they should be spending their time to best impact the business. This covers event management to solve immediate issues, resource allocation and through reporting performance based on the data of applications, infrastructure, networks and from third-party platforms. BSM includes powerful analytics that gives IT the means to prepare, predict and pinpoint by learning behavior and analyzing IT data forward and backwards in time using Big Data Analytics applied to IT Operations.

Critical steps to a successful IT operations analytics strategy

It’s easy to get overwhelmed with data. In order to ensure IT management has the right information to solve a problem, you need to collect everything and face down the challenges of storing, managing and using it all. Here are four components you need in an analytics strategy to turn all that data into valuable insight

I know what server did last second

 

Second counts.png

 

 

 

In the era of DevOps and cloud computing, the ability to avoid downtime and monitor your application infrastructure performance in real time can be a real competitive edge.

Find out how you can

Analytics in IT Operations — EMA Research white paper reveals 4 key qualities of new platforms

Operations Intelligence.pngBig data and analytics play a growing role in how IT organizations are trying to improve efficiency and value in the services they deliver to the business. A key focal point of this widespread trend is the Operations Center, and EMA Research offers some insightful analysis about the class of analytics tools that have emerged and the trends that are driving them.

 

With written contribution from Andrew Wahl

How to look “under the hood” of your private cloud

Hood.jpg

Standard monitoring tools are great to provide the big picture of your datacenter health , but they lack the granularity you need for root cause analysis.

 

 

Here is how HP solves this problem, so you can get deep insights into your cloud systems performance in real time.

Using Operations Analytics to solve IT problems

A year ago I spoke with a customer about an application that was having response time issues. The IT staff could not figure out what was causing the issues with this application –even after spending hours investigating the issue.

 

Keep reading to find out what the issue was and how the team resolved it.  You can also learn how the issue could have been resolved weeks earlier if Operations Analytics had been installed.

Operations Analytics customers say IT Operations and the business must be aligned to achieve success

We have spent our week here at HP Discover in Barcelona speaking with customers.  From what Operations Analytics customers said at the BSM Customer Advisory Board and the Special Interest Groups, IT and Business are not always communicating effectively.

 

Continue reading to find out how IT can learn to speak with business decision makers and convey their messaging to accomplish their mission.

 

Guest Post Yair Horovitz , Chief Functional Architect - Big Data Analytics 

Needles in IT Haystacks or How to take up to 80% cost out of IT Operations

The costs associated with shadow IT are hiding around the corner—are you on the lookout?  As an IT professional, you may not think that shadow IT has costs associated with it. The issue for your team is the maintenance of these products. If something goes wrong, your team is the first line of response that they call.

 

The problem is that you may not have full visibility into these elements or those of BYOD. Unfortunately they do impact your environment. Keep reading to find out how to can gain clearer vision into all of these elements and learn how to manage them better.

Operations: application performance sucks, what do I do now?

by Michael Procopio


In my IT days (it has been a while) as still happens today this is the question many have asked. It’s more complicated today, applications are more distributed now. However, you still have to go through the triage process. The topic these days is named “Application Performance Management” or APM.


APM has two parts, the traditional looking at infrastructure resources and measuring performance from the end user perspective.


You probably detected this problem in one of two ways. You are ahead of the curve and have end user monitoring in place or the user called the help desk to complain.


A typical web based application today uses a web server, application server and a backend, typically a database. Though the backend might actually have multiple parts if service oriented architecture, (SOA) is used. Good news, Operations Manager, agent based, and SiteScope, agentless, will provide status on condition of those servers.


These tools can also look at how packaged applications are doing on the server. Oracle, WebSphere, MS Exchange, MS Active Directory, to name a few, can be monitored by either Operations Agents or have SiteScope templates (a SiteScope template is a prepackaged set of monitors). These tools might point to something as detailed as database locks being far higher than normal and beyond the current setting on the database. A quick parameter change might fix this.


Next, we have the code. We hope this isn’t the case because this typically moves the problem from operations to development. However, Operations is still responsible for pinpointing the problem area. This is typically the domain of application support and in some organizations that’s inside Operations, in others a different group.


Here Business Transaction Management (BTM) tools can help. BTM manages from a transaction point of view. BTM includes transaction tracing. TransactionVision and Diagnostics work in a complimentary fashion to give you the next level of detail although each is usable separately. TransactionVision traces individual critical transactions (as you define them) through multiple servers; it gives you information on a specific transaction including the value of the transaction.


Diagnostics provides aggregate information on all transactions in a composite application giving you timing information. It can pinpoint:


· where time is spent in an application; either processing data or waiting for a response from another part of the application.


· the slowest layers.


· the slowest server requests which are the application entry points.


· outliers to help diagnose intermittent problems.


· threads that may be contributing to performance issues.


· memory problems and garbage collection issues.


· the fastest growing and largest size collections.


· leaking objects, object growth trends, object instance counts, and the byte size for objects.


· the slowest SQL query and report query information.


· exception counts and trace information which often go undetected.


TransactionVision and Diagnostics also integrate with Business Availability Center, which means you can start with a topology view and drill all the way down to find the status of the most valuable transaction running through you systems.


You can manage what you can’t measure. So what do I do now? If you are properly instrumented the problem will show itself. If you don’t find something you can fix, you can tell the app developers where they need to look to fix the problem.


   


Related Items:


· End User Monitoring


· Operations Manager


· SiteScope


· SiteScope Administrator Forum


· TransactionVision


· Diagnostics


· Business Availability Center





Fighting or friendly, Problem Isolation and OMi

by Michael Procopio


In the post  Event Correlation OMi TBEC and Problem Isolation What's the Difference, my fellow blogger, Jon Haworth, discussed the differences between TBEC and Problem Isolation. To be consistent, I'll use the acronyms PI for Problem Isolation and TBEC to refer to OMi (Operations Manager i series) Topology Based Event Correlation.


Briefly, he mentioned that TBEC works “bottom up”, that is starting from the infrastructure, with events. PI works “top down”, that is, starting from an end user experience problem, primarily with metric (time series) data.


Jon did an excellent job describing TBEC; I’ll do my best on PI because like Jon I have a conscience to settle.


Problem Isolation is a tool to:


1. automate the steps a troubleshooter would go through


2. run additional tests that might uncover the problem


3. look at all metric/performance data from the end user experience monitoring and all the infrastructure it depends


4. find the infrastructure metric the most closely matches the end user problem using behavior learning and regression analysis techniques (developed by HP Labs)


5. bring additional data such as events, help/service desk tickets and changes to the troubleshooter


6. allow the troubleshooter to execute Run books to potentially solve the problem


Potentially the biggest difference in the underlying technology is that Problem Isolation does not require any correlation rules or thresholds to be set for it to do the regression analysis to point to the problem. Like TBEC, it does require that an application be modeled in a CMDB.


An example: Presume a situation with a typical composite application - web server, application server and database. No infrastructure thresholds were violated; therefore, there are no infrastructure alerts. Again, as mentioned in the previous post, end user monitoring (EUM) is the back stop. EUM alerts on slow end user performance, now what?


Here is what Problem Isolation does:


1. determines which infrastructure elements (ITIL configurations items or CIs) support the transaction


2. reruns the test(s) that caused the alert – this validates it is not transient problem


3. runs any additional tests defined for the CIs


4. collects Service Level Agreement information


5. collects all available infrastructure performance metrics (web server, application server, database server and operating systems for each) and compares them to the EUM data using behavior and regression analysis



Problem Isolation screen show performance correlation between end user response and SQL Server database locks


-------------------------------------------------------------------------------------------


6. determines and displays the most probable suspect CI and alternates


7. displays run books available for all infrastructure CIs for the PI user to run directly from the tool


8. allows the PI user to attach all the information to a service ticket, either existing or create a new one


Another key differentiator of OMi/TBEC and PI is the target user. There is such a wide variance in how organizations work that it is hard to name the role but let me do a brief description and I think will be able to determine the title in your organization.


There are some folks in the organization whose job is to take a quick look (typically < 10 minutes, in one organization I interviewed < 1 minute) at a situation and determine if they have explicit instructions on what to do via scripts or run books. When they have no instructions for a situation they pass it on to someone who has a bit more experience and does some free form triage.


This person might be able to fix the problem or may have to pass it on to a subject matter expert, for example if they believe it is an MS Exchange problem to an Exchange admin. It is this second person that Problem Isolation is targeted at. This is helping automate her job, reducing what might take tens of minutes to hours and performing it in seconds. If it ends up she can’t solve the problem it automatically provides full documentation of all information collected. That alone might take someone five minutes to write-up.


OMi’s target is the operations bridge console user. Ops Bridge operators tend to be lower skilled and face hundreds if not thousands of events per hour. Jon described how OMi helps them work smarter.


TBEC and Problem Isolation both work to find the root cause of an incident but in different ways. Much like a doctor might use an MRI or CAT scan to diagnose a patient based on what the situation is, TBEC and Problem Isolation are complementary tools each with unique capabilities.


Problem Isolation will not find problems in redundant infrastructure that OMi will. Conversely, OMi can’t help with EUM problems when no events are triggered, where Problem Isolation will.


We know this can be a confusing area. We welcome your questions to help us do a better job of describing the difference. But these two are definitely friendly.


For Business Availability Center, Michael Procopio


Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps


Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.


Related Items



  1. Advanced analytics reduces downtime costs - detection

  2. Advanced analytics reduces downtime costs – isolation

  3. Problem Isolation page

  4. Operations Manager i page

Announcing a New BSM Solution Offering for Virtualization

by Michael Procopio


The new offerings include enhancements to HP
Server Automation
, HP
Client Automation
, HP
Storage Essentials
, HP
Network Automation
, and HP Operations
Manager
software.


According to a recent report from
Gartner(1), "Virtualization’s impact on the overall IT industry has
been dramatic, and virtualization will continue to be the leading catalyst for
infrastructure and operations software change through 2013. Organizations are
looking at ways to cut costs, better utilize assets, and reduce implementation
and management time and complexity."


Although virtualization is often adopted to help reduce capital expenditures,
it can trigger increased management expenses and lead to more pronounced
organizational silos. The new HP offerings bridge all the physical and virtual
data center silos through management and automation. This reduces complexity and
ultimately management costs.


"HP business service automation software helps us eliminate manual,
error-prone tasks by automating server lifecycle management, including
provisioning multiple operating systems, software installation, deployment of
patches, configuration management and audits," said Ron Cotten, senior manager
IT OSS Engineering, Level 3 Communications, a leading international provider of
voice, video, and data communications services. "With HP Server Automation, we
are able to patch over 1800 servers in 24 hours, which helps us reduce scheduled
downtime."


The updated HP business service offerings help you:


 


· Increase administrator effectiveness with HP
Operations Manager for virtualization
by monitoring the availability and
performance of all virtual and physical assets through a common dashboard.


· Reduce the risk of downtime with HP Network Automation, which for the
first time gives the network administrator control of the VMware vSwitch in
addition to the physical network environments.


· Provision the right amount of storage to keep applications performing
properly without overspending on excess storage with HP
Server Automation
, the first solution in the industry that gives server
administrators this capability.


· Reduce problem resolution times with HP
Storage Essentials
Performance Edition by quickly identifying,
troubleshooting and reporting performance metric related trends in physical and
virtual environments.


 


“To make virtualization cost effective, customers must minimize operating
expenses and have seamless management of infrastructure silos,” said Erik
Frieberg, vice president of Product Marketing, Software & Solutions, HP.
“Our newly enhanced HP business service offerings help customers manage all
aspects of the physical and virtual application infrastructure to unlock the
true promise of virtualization.”


HP Software Professional Services provides solution consulting services to
accelerate the value of business service automation and business service
management software investments.


HP
Server Automation
, HP
Client Automation
, HP
Storage Essentials
and HP
Network Automation
are available now. HP Operations Manager Virtualization Smart
Plug-In
will be available next month.


 


 

HP Software Universe – Mainstage Andy Isherwood


Andy Isherwood VP, Support & Services
kicked off Mainstage.


There are four key areas shown in the picture above. HP announced this week its  IT Financial Management offering. Andy likened ITFM to an ERP system for IT. Information Management magazine wrote an article on HP ITFM.


HP has had offerings in IT Performance Analytics and IT Resource Optimization for awhile. HP Cloud Assure was announced was announced in May 2009, HP Unveils “Cloud Assure” to Drive Business Adoption of Cloud Services


Some key points from his opening remarks:



  1. Prepare for coming out of the
    recession when cutting costs and innovating.


  2. Best in class means being good at all
    four - aligning to the business, taking out costs, increasing efficiency and
    consolidation.


  3. Jetblue, Altec and T-Mobile were the
    winners of the HP Software Award of Excellence.


  4. As an example of the quick ROI
    companies can get, Altec produced 10% application downtime reduction, 20% faster
    response time, 15% increase in customer satisfaction and a 300% improve
    application transaction time in 6 months.


  5. Last year we were HP Software, this
    year HP Software and Solutions. This was the combining HP Software with HP
    Consulting and Integration. The net result increased our delivery options. In
    addition to offering software for in-house use, HP now has EDS, SaaS and
    continues with it Partners


  6. HP SaaS business is seven years old
    this year and has 650 customers.


You can read other coverage of HP Software Universe in the ITOpsBlog. There are a variety of Twitter accounts
you can follow:


HPITOps  – Covers BSM, ITFM, ITSM, Operations and
Network Management


HPSU09  – show logistics and other
information


HPSoftwareCTO


informationCTO


HPSoftware


BTOCMO – HP BTO Chief Marketing Officer


as well as the Twitter hashtag #HPSU09


For HP BSM, Michael Procopio


 

HP Software Universe - day 1

by Michael Procopio


 


Today was the first day of Software Universe. I had customer meetings all day today. Here are some interesting items from my conversations.



  1. Most said budgets were down in 2009 and will be flat to down in 2010. But a few who were related to government stimulus said theirs will be up.

  2. Co-sourcing and outsourcing continue as ways to reduce costs

  3. A few were focusing on asset management with the express purpose of getting rid of things in the environment they don’t need anymore. They know they are out there but they need to find them first.

  4. Most customers I spoke to said they keep aggregated performance data for 2 years the range was 18 months to 5 years.

  5. There was an interesting discussion about the definition of a business service versus an IT service. The point being made was a business service by definition involves more than IT. While I agree this is a good point, I think the IT industry has focused on business service as a way to say - “I’m thinking about this IT service in the context the business thinks about it not just from my own IT based perspective”

  6. A number of customers have or are about to implement NNMi. If this is something you are interested in check out the NNMi Portal

  7. Many customers are moving to virtualized environment highest percentage I heard was 70%. Another customer forces all internal developers to deliver software as a virtual image.

  8. Another topic was how to monitor out tasked items. For example, some part of what you offer is delivered by a third party - how do you make sure they are living up to your standards. Two methods I heard were 1/ use HP Business Process Monitor 2/ get the 3rd party to send you alerts from their monitoring system.

  9. On the question does your manager of managers send back data to sync the original tools 1 did, 1 didn’t. For the one who did it was part of a closed loop process.

    • Monitor tool finds problem send alert to MOM (Manager of managers).

    • MOM send event ID to monitoring tool

    • Subject matter expert uses monitoring tools to diagnose problem

    • Once diagnosed updates monitoring tool which updates MOM




A very productive day for me. I hope some of this is useful information to you.


For additional coverage my blogger buddy Pete Spielvogel is also here and beat me to the first post. You can read his posts at the ITOps Blog.


There are a variety of Twitter accounts you can follow as well as the hashtag #HPSU09


HPITOps – Covers BSM, Operations and Network Management


HPSU09 – show logistics and other information


HPSoftwareCTO


informationCTO


HPSoftware


BTOCMO – HP BTO Chief Marketing Officer 


 


For HP BSM, Michael Procopio

BSM at HP Software Universe

by Michael Procopio


 


HP Software Universe is next week, 16-18 June, in Las Vegas. Business Service Management (BSM) will be well represented.


In the Business Transaction Management area there are 13 sessions. Most of them are lead by customers. The sessions are listed below.


In the Network Management track Aruna Ravichandran is speaking in three sessions, you can see information on those at her post HPSoftware Universe/HP Technology Forum (HPTF) - Network Management sessions. The rest of the track is listed in the post Network Management at HP Software Universe.


Amy Feldman, Dennis Corning and Peter Spielvogel the ITOps bloggers has covered a number of the sessions in the Consolidated event and performance management. Here are a list of the posts:



 


Business Transaction Management Track













































































































Session ID Title Presenting company
1114 Confessions of a product manager: get the real scoop on the latest HP Business Availability Center HP
1165 The MITRE Corporation: higher operational effectiveness at lower cost through automated alert management MITRE Corporation, AlarmPoint
1233 Key decisions and practical techniques in configuring business transaction management
1236 Real User Management: know how your TCP/IP applications perform for your users HP
1267 Using HP Business Availability Center to analyze and triage application and infrastructure anomalies and problems BCBS of Florida
1303 Sodexo: partnering with HP Software-as-a-Service to ensure critical e-business application performance and availability Sodexo
1342 Wrigley: HP Business Availability Center deployed on Software-as-a-Service yields big improvements in IT monitoring without increasing staff Wrigley
1360 Lockheed Martin: deploying HP Business Availability Center in a virtual environment and forwarding alerts through an iPhone Twitter-based application Lockheed Martin
1363 DIRECTV: an HP Business Availability Center and HP operations implementation DIRECTV
1401 Liberty Life: taking the fast track to implementing HP Business Availability Center and gaining business value in 6 months Liberty Life
1425 Sentara Healthcare: improving the availability of critical business services and fixing IT problems before they impact customers Sentara Healthcare
1436 Lockheed Martin: practical advice for configuring and operating HP End User Management solutions Lockheed Martin
1452 Vale: deploying HP Business Availability Center solutions to monitor applications and systems and to help ensure availability and performance Vale

 You can get the details of all the BSM sessions at the HP Software Universe Track Session Catalog.


I hope to see you there, but if you can’t make it we will be doing follow-up posts. You can also follow on Twitter, the hashtag is #HPSU09. There are already a number of Tweets and the show hasn’t started yet. The Twitter account for the show is HPSU09, if you’d like to follow us. Or visit the HP Software Universe Facebook page.


For the Business Availability Center, Michael Procopio


 

BSM Evolution: Small Enterprise Example

My previous BSM evolution postings focused on mega-corporations and large IT organizations with a myriad of personas.  In this post, I will contrast the experience of a relatively small IT shop of roughly 30 full time IT operations personnel.


Back when the economy was cooking along, an up and coming commercial construction company grew right out of their business model.  Historically, they utilized a decentralized model, setting up and staffing a stand-alone onsite operation for each new project. This model was excellent at delivering customized project support, but lacked scalability and leverage; with remote site spin-up slow and error prone.


From an IT perspective, the CIO realized they needed to, in his words, "Consolidate and professionalize the IT operations", with the following goals:



  • 1. Improve quality of service and experience for worksite users & applications

  • 2. Contain IT costs and efficiently scale current IT personnel to meet growth

  • 3. Improve speed, accuracy, and agility of spinning up new project worksites

Key Personas:


CIO



  • Many years of commercial construction experience

  • Personally drove IT consolidation / professionalization strategy and roadmap

  • Directly engaged in evaluating and selecting the solution vendor/consultant

VP of IT



  • "Co-pilot" for CIO on strategy, drove project deployment and vendor engagement

Subject Matter Experts (SME)



  • One for performance and availability tools / architecture

  • One for service management process workflow and automation (helpdesk)

Two Key Parallel Evolution Paths:


Path A:  Performance, availability, and quality of experience monitoring


Step 1: -Deployed synthetic end-user / application monitors, agentless remote site infrastructure monitoring, and general WAN/LAN management


             -Basic service experience reporting, and per-site performance dashboards


Step 2:  -Enterprise infrastructure fault/performance (agent based system, OS, DB)


             -Central "IT Command Center" event console with trouble ticket integration


Step 3:  -In-depth application management modules (exchange, SAP)


             -Advanced network services (route analytics, performance)


Path B:  Service management process workflow and automation


 Step 1:  -Single call/request center organization established


              -Incident management (utilized pre-packed ITIL module)


 Step 2:  -Knowledge management process, analytics and automation modules


 Step 3:  -Configuration and change management process/automation


              -Service Level Management definition and basic reporting


An Uncommon Sequence of Evolution Steps


Notice the interesting order of the steps.  The CIO dictated that the performance monitoring path start with remote site end-user / application experience monitoring.  The original roadmap proposed by the system integrator recommended starting with basic data center tools, advancing through central event console, then application and database management, and finally end user experience.  This is a traditional evolution path, but the CIO was adamant that, "what happens at the remote work-sites IS the business".  So, he wanted an immediate awareness of remote site experience to drive the design of every step in the roadmap. 


There was a similar "cultural" direction from the CIO on the service management workflow path.  Again, the CIO insisted that Knowledge management be moved up in the evolution before configuration, change, and service level management.  Typically, significant knowledge management execution is viewed as "icing on the cake" by most organizations, and only implemented after all the other core ITIL processes.


This CIO believed that analyzing and formalizing knowledge learned from successes and failures of spinning-up remote sites and dealing with issues was the best early investment. This approach immediately became part of the standard IT culture, and played a significant role in guiding change and configuration management process definition.


The CIO's Project-Based perspective


This CIO is indeed very ITIL savvy, but I think living and breathing the commercial construction business had a significant impact on his choice of system integrators. During the bidding process for the ITSM/BSM contract, it came down to three competitors in a direct "shoot-out". System integrator number one and two brought product and ITIL experts to the shoot-out, concentrated very heavily on features and functions, and gave a fixed-price bid of 200 deployment days. 


System integrator number three brought a project manager to the shoot-out, and changed 75% of the discussion to, "here is how we will navigate the project and be successful". Can you guess who won? It shouldn't be news to anyone that a CIO's background alters the decision criteria, or the roadmap vision.... But it is always interesting to observe it in action.  Maybe I will write a post about that someday


Conclusion


This IT organization is relatively small, so the decision making process and personas are greatly simplified compared with the large corporations previously analyzed. Despite the CIO's unique influence on approach and deployment sequence, in the end, the same fundamental truths of BSM/ITM evolution apply.... Just on a different scale, agility and timeframe.    


Bryan Dean - BSM Research



 Tweet this! 


Related Items



 


 


 

BSM Evolution: The CIO/Ops Perception Gap

 


There are many potential culprits for why IT organizations struggle to make substantive progress in evolving their ITSM/BSM effectiveness. A customer research project we did a few years ago offered an interesting insight into one particular issue that I rarely see the industry address. The research showed that most CIO’s simply had a different perception – when compared to their IT operations managers- of their IT organization’s fundamental service delivery maturity and capability. This seemingly benign situation often proved to be a powerful success inhibitor.


 


The Gap:


A substantial sample size of international, Global 2000 enterprise IT executives participated in the study. When asked to prioritize investment priorities on a broad range of IT capabilities, we saw a definite gap. IT Operations managers consistently ranked, “Investing to improve general IT service support and production IT operations” in their top 1 or 2 priorities, where CIO’s ranked this same capability much lower as a priority 6 or 7.


 


The Perception:


When pressed further, CIOs believed that the IT service management basics of process and technology were already successfully completed, and the CIO’s had mentally moved on to other priorities such as rolling out new applications, IT financial management, or project and portfolio management.


 


Most of the CIOs in the study could clearly recall spending thousands of dollars sending IT personnel to ITIL education, and thousands more purchasing helpdesk, network, and system management software. Apparently, these CIO’s thought of their investment in service operations as a onetime project, rather than an ongoing journey that requires multiple years of investment, evolution, reevaluation, and continuous improvement.


 


IT operations managers on the other hand- clearly had a different view of the world. They were generally pleased with the initial progress from the service operations investments, but realized they were far from the desired end state. The Ops managers could plainly see the need to get proactive, to execute advanced IT processes and more sophisticated management tools, but could not drain the proverbial swamp while fighting off the alligators.


 


The Trap:


We probed deeper in the research, diligently questioning the IT operations managers on why they didn’t dispel the CIO’s inaccurate perception. In order to secure the substantial budget, these Ops managers had fallen into the trap of over-promising the initial service management project’s end-state, ROI and time to value. (I wouldn’t be surprised if they had been helped along by the process consultants and software management vendors!)


 


These Ops managers saw it as “a personal failure” to re-approach the CIO and ask for additional budget to continue improving the IT fundamentals. Worse yet, they had to continually reinforce the benefits from the original investment so the CIO didn’t think they had wasted the money. So, the IT operations staff enjoyed the result of reactively working nights and weekends to meet business’ expectations, and make sure everyone kept their jobs. Meanwhile, the CIO’s slept well at night thinking, “Hey, we are doing a pretty darn good job”, but faced the next day asking, “Why are my people burnt out?” A vicious cycle.


 


Recommendation through Observation:


Im not wild about making recommendations since I merely research this stuff… not actually perform hands-on implementation. Instead, I will offer some observations of best practices from companies who appear to be breaking through on BSM, lowering costs, raising efficiency and improving IT quality of service.


 



  1. Focus on Fundamentals: It is boring and basic, but absolutely critical to continually look for ways to improve the foundational service management elements of event, incident, problem, change, and configuration management. Successful IT organizations naturally assume that if they implemented these core processes more than 3 years ago, they likely need to update both technology and process. If FIFA World Cup Football clubs and Major League Baseball teams revisit their fundamental skills each and every year, why wouldn’t IT?

 



  1. Assume a Journey: IT leaders who develop a step-wise, modular path of realistic projects that deliver a defined ROI at each step have the best track record of securing ongoing funding from the business. The danger here is defining modular steps that are so disconnected and silo’d, that IT never progresses toward an integrated BSM/ITSM process and technology architecture. This balance continues to be one of the most difficult to manage.

 



  1. Empowered VP of IT Operations: The advantages of a CIO empowering a VP of IT operations and holding them accountable for end-to-end business service has been discussed in previous posts. The practice of having a strong VP of operations who has executive focus on service operations and continual service improvement, while having end-to-end service performance responsibility does appear to be a growing trend and success factor.

 



  1. Focus on the Applications: In the same research study that showed the perception gap on, “Investing to improve general IT service support and production IT operations”, there was consistent agreement on, “Investing to improve business critical application performance and availability”. The CIO’s, Ops Managers and Business Relationship managers all ranked this capability as a top 1 or 2 priority.

 


Successful BSM implementations focus on the fundamentals of process and infrastructure management, but do so from a business service, or an application perspective. This approach not only enables an advantageous budget discussion with the business, but it also hones the scope and execution of projects.


 


It is difficult to assess the relative impact of this CIO/IT Ops perception gap, considering the wide variety of challenges that IT faces. But hopefully, this post gives you something to consider when assessing your own IT organization’s situation and evolution.


 


Let us know where your organization fits – please take our two question survey (two demographics questions also). We’ll publish the results on the blog.


 



  • Describe the perception of your IT's fundamental service delivery process

  • How often does your IT organization significantly evaluate and invest to update your fundamental IT process

 


Click Here to take survey


 


Bryan Dean – BSM Research

Webinar announcement: "Decrease IT Operational Costs by Accelerating Problem Resolution"

In a recent post, I talked about  “from user experience monitoring to user experience management”.  


Related to this, a recent Forrester study found that 74% of problems with business services are reported by the end users through the service desk; not reported by infrastructure management tools. The same survey found that an average of six service desk calls are needed to identify the problem owner for a top-down performance problem.  


 


What is therefore needed if we want to increase IT Ops efficiency and stop using our customers as the most expensive monitoring devices there are, is to proactively monitor customer experience before our customers do and have the tools to quickly and accurately pinpoint the cause of business service performance problems.  


Senior Enterprise Management Associates (EMA) Analyst Liam McGlynn and my colleague Sanjay Anne are conducting a webinar on this topic on March 19. The webinar is entitled “Decrease IT Operational Costs by Accelerating Problem Resolution”.


 


For more details and to register, please go to: http://www.enterprisemanagement.com/research/asset.php?id=1127.


 


 

BSM Evolution Paths: Auto Industry Sample

In the last post, Bryan Dean, our research expert in the BSM team, outlined the different ways in which customer evolve towards Business Service Management. In the next few posts, Bryan will give an example of each of the different types of evolution. Over to you Bryan ....
_______
About three years ago, the business division managers of a multinational automobile manufacturing company planned a bold transformation of their distribution network to leapfrog the competition.  They enthusiastically laid out a roadmap for business process innovation and aggressive customer/dealer satisfaction initiatives.  

Only one real problem; the CIO knew that building, rolling-out, and operating the underlying IT for this future business vision exceeded their current capabilities.  The CIO eventually had to raise the red flag and explain to the executive committee why IT was the bottleneck.  Ouch, not a good day.

 

In the previous post BSM Evolution Paths:  Samples and Observations, we talked about five common evolution paths, the organizational and persona dynamics of an Automated BSM/ITSM journey.  In this post we will overview a specific example.


 


To be fair, the CIO spent years driving significant investment in process, tools and the organization.  Let’s look at a subset of key personas and BSM/ITSM foundation: 

 

Director of Infrastructure (reporting to the VP Global IT Ops):



  • Enterprise-class central event/performance platform and console

  • WAN/LAN network management platform

  • Basic, component level performance and availability reporting

  • Dozens of vendor-specific configuration and admin tools

Director of Service Management (reporting to the VP Global IT Ops):



  • Global, consolidated helpdesk/service desk

  • Well defined and automated incident process; basic level problem, configuration, and a manual change process

Director of Applications (development, test & level 3 support.  Reports to business divisions):



  • Suite of pre-production stress-test quality and performance tools

  • End-user  and application performance/diagnostic tools (test environment)

The Key Evolution Steps


Step 1:  CIO empowers and holds the VP of Global IT Operations (VPITops) accountable for end-to-end business service responsibility.  Imagine the panic on his face!  VPITops launches key lieutenants on quick gap analysis.

 

Step 2:  The VPITops needed a quick win.  He believed that visually demonstrating and reporting performance and availability from a business service perspective -versus an infrastructure perspective- would be a catalyst for driving “aligned” IT behavior.  The current network and infrastructure products didn’t have this capability, so VPITops leveraged the tools already proven by the application test and level 3 support team.


VPITops established a new team within Operations (parallel to infrastructure event management) to own and run the end-to-end business service visibility/accountability solution.  Integration was established between the two teams and tools.

 

Step 3a:  VPITops took his new business service visibility/accountability tool (in dashboard/report form) to key business division managers, and established a business relationship management function.  This converted the conversation from anecdotal complaints, to measurable service levels.  The CIO had tangible proof of progress.

 

Step 3b:  While engineering step 2, the Tools and Process Architect realized they needed a better means of discovering and maintaining the IT/Business service models.  Their infrastructure environment was shared, complex and dynamic enough that static service models were not effective, so they brought in an application dependency mapping technology.  This success spawned a serendipitous benefit to another team in step 4a.

 

Step 4a:  The application quality/test and release team realized the service model could be utilized in the service transition process.  They previously had several very painful episodes of moving complex applications from test into production.  With an accurate, up to date service model of the production environment they could better identify dependency issues before roll-out.  Speed and accuracy...  Happy CIO.

 

Step 4b:  The Director of Service Management and the architect evaluated how to federate the data between the application dependency mapping service model and the CI configuration data in the helpdesk.   The software vendor provided a federation / reconciliation adaptor, so the helpdesk was able to leverage the CI relationships and operate off a “single version of the truth” (sounds eerily like an ITIL V3 CMS!). 

 

Near Term Roadmap


·         Automate change/configuration workflow and provisioning


·         Upgrade/replace enterprise event and performance console to leverage service model for root cause analysis and business impact assessment


·         Apply business service relationship management to additional business divisions


·         End-to-end visibility of composite MQ application business transactions   

 

The Verdict of the Journey so far


The CIO still has a job, and has a funded roadmap.  One might ask why they didn’t start with step 4b, and establish the CMDB and service model first?  Well, the CIO was on the hot seat, and they were concerned about getting bogged down in an enterprise-wide CMDB architecture project. 

 

This exemplifies the unpredictable and unique nature of evolution paths.  More can be said about the delicate balance between tops-down guidance, and fostering organic innovation from within the ranks of IT.  In future posts, I will discuss and analyze this further, as well as introduce other examples.

There are a number of ways of populating the service dependency map

 


In a post two weeks  on this blog, I listed all the ways that we use service dependency maps (model-based event correlation, service impact analysis, top-down performance problem isolation, SLAs, etc).  What can be used to discover service dependency information?


 


OperationsCenter Smart Plug-ins (SPIs) now discover to the CMDB


If you're using the agent-based side of OperationsCenter (OpC), then each managed node will have an agent on it. You can put a smart plug-in (SPI) onto that agent. SPIs have specialized knowledge of the domain they are managing. There are many SPIs for all kinds of things from infrastructure up to applications like SAP. Many of the SPIs discover (and continue to discover) the environment they are monitoring. This is agent-based discovery using all the credentials you've already configured into the OpC agent.




The OMi team are working on putting SPI-based discovery information into the HP CMDB (the Universal CMDB or uCMDB).


 


Agentless monitoring populates the uCMDB


If you have agentless monitoring (HP SiteScope) this will populate the uCMDB too (as of SiteScope version 10).




Whatever SiteScope monitors you have configured will send their configuration information to the uCMDB. So, if you're monitoring a server with a database on it, all the information about the server and its database will be sent to the uCDMB.


 


Network Node Manager populates the uCMDB


As of the latest version of Network Node Manager (NNMi 8.10), discovered network end-points are also put into the uCMDB. "Network end-points" are anything with a network terminator - network devices, servers, and printers. NNMi provides no service dependency information, but it does provide an inventory of what's out there.




This inventory discovery is useful for rouge device investigation - noticing an unknown device, creating a ticket to the group responsible for that type of device so they can look into it.


 


Standard Discovery


Our Standard Dependency Discovery Mapping product (DDM-Standard) will discover your hosts for you. This also discovers network artifacts (but, see NNM discovery above - if you have NNMi, this is a more detailed network discovery mechanism).


 


Advanced Discovery


Advanced Dependency Discovery Mapping will discover storage, mainframes, virtualized environments, LDAP, MS Active Directory, DNS, FTP, MQSeries buses, app servers, databases, Citrix, MS Exchange, SAP, Siebel, and Oracle Financials.




You can also create patterns for top -level business services and DDM-Advanced will discover those too.


 


Transaction Discovery


Our Business Transaction Management product, TransactionVision,  deploys sensors to capture application events (not operational events) from the application and middleware tiers. These sensors feed the events to the TransactionVision Analyzer which automatically correlates these events into an instance of a transaction. TransactionVision also classifies the transactions by type - bond trade, transfer request, etc. Thus, TransactionVision is discovering transactions for you.




TransactionVision puts this transaction information into the CMDB. In other words, the CMDB doesn't just know about "single node" CI types like servers, it also knows about flow CI types - transactions.




Also, if the CMDB notices that the transaction flows over a J2EE application, it links the transaction to information in the CMDB about this J2EE application - the transaction step and the J2EE app are now linked in the model. .


__________


 


By the way, my colleague Jon Haworth has just posted on the value of discovery in the realm of Operations Management at ITOpsBlog (28th January, "Automated Infrastructure Discovery - Extreme Makeover").

Answers to questions on "what's new in Business Availablity Center 8.0?"


I recently mentioned about a whats new webinar conducted on BAC v 8.0. You can now access this on-demand webinar at


https://h30406.www3.hp.com/campaigns/2008/events/sw-01-20-09/index.php?rtc=3-2CDASIY


 


Here are some of the questions which came up during the live webinar.


 


Q: When will 8.0 be available?


A: The 8.0 release will be made available the first week of February


 


Q: How will the new modeling changes affect my current custom views?


A: There are no more instance views, it’s just views and custom perspectives that provide the content in the view, the upgrade for most customers should be straightforward, unless they have changed the model, created new CI types with custom links or are heavily using pattern views with impact analysis, correlation rules and alerts.


 


Q: How about integration with HP Operations Manager? Can we leverage our current HPOV infrastructure monitoring capabilities and marry data with BAC application monitoring?


A: Yes with HP problem Isolation we have support of OM (Operations Manager) through event correlation to application problem/ anomaly start time.


 


Q: Does v 8.0 support oracle 11g platform


A: Yes with v 8.0 is it supported.


 


Q: I was told that the DDM portion of the new 8.0 can discover WebLogic 10.x iis it true?


A: Yes with v 8.0 is it supported.

Can I get away without using discovery?

When I was at our European HP Software event before Christmas / The Holidays, I spent a good deal of time talking to people about our new product releases and the future of BSM. One customer looked a little worried and said, "wow - you seem to rely on discovery a lot".


I guess there are two things to say in answer to that observation. The first is "yes - because we rely on the service hierarchy model a lot". And the second is, "but there are a number of different types of discovery - and a number of them you already have".


 


So, in a two part post, I thought I'd answer that observation more comprehensively. So, let's first look at how we use inventory and service hierarchy information in the management of service health (and thanks Jon Haworth from the OperationsManager team for his significant help on this post):


 



  • It helps with administering the monitoring deployment of the managed environment. It tells us what is out there, what we need to manage, what has disappeared, and so on. This only requires discovery of the infrastructure inventory – "tell me what servers exist" (unless everything is virtualized, in which case it needs a lot more. The OperationsCenter team has posted on the new virtualization SPI recently at ITOpsBlog. This SPI discovers, and more importantly, continues to discover, virtualized environments).

 



  • It helps OMi to understanding the stream of events which are being detected in the infrastructure and applications. The hierarchy of the monitored items ("configuration items" or CI's) allows OMi to tell us which events are causal events and which are symptoms – what do we need to work on and what can we ignore. I talked about how OMi does this in a post last year.

 



  •  It allows all parts of the BSM stack perform service impact analysis. This is where events are related to infrastructure and applications and their impact or potential impact on the services above in the hierarchy is established. We can then use this impact information to prioritize the events.  Service impact analysis requires a model of the hierarchy of CI's and services.  Maintaining the service hierarchy manually is untenable -- things just change too rapidly for humans to keep up.  

 



  • When a disk has a set of read/write errors, is that catastrophic? If it's a single disk, then yes - the infrastructure element is in trouble. If it's part of a RAID array, then no -  provided the rest of the array is OK.  If we know the type of CI that we seeing events against, we can make better decisions about its true health.

 


This is also a new feature in OMi: when CI's are discovered we know their type. OMi ships with a database of health indicators for each CI type. For example, for single disks, it's a problem if the disk gets bad errors; if it's a RAID array, then provided a high percentage of the other disks are OK, this is not a serious problem; and so on.


 


This feature makes the calculation of the true health of CI much easier. You don't need to define a set of propagation rules. OMi uses the discovered CI type information and it's lookup table to figure out propagation itself.


 


This all ties into a new feature in OMi called "Health Indicators". Jon Haworth has promised to post on this on his team's blog at the OperationsCenter blog


 



  • Our top-down performance Problem Isolation software needs to understand the service hierarchy on which the end user application rests. For example, if I have a web user interface, I need to understand what services that user interface depends on. As I discussed in a post last year, problem isolation uses statistical correlation analysis to suggest the likely cause of such top-down performance problems.

 



  • We need the service hierarchy for defining SLAs. I may define a compound SLA that depends on a number of OLAs and a top-level measured SLA. The modeling user interface for this and the subsequent off-line SLA calculation is done based on the service hierarchy.

 


In the second part of the post, I'll talk about all the things that now populate the host inventory and service dependency map.  Hint: if you have SPIs, you'll like what we have to say :-)


 


Mike Shaw

Announcing Business Availability Center 8.0

In a post last year, I talked about how to move from user experience monitoring to user experience management, you need to be able to figure out what is the cause of a measured user experience problem, like slow on-line  check-in times.  I talked about a tool we have called Problem Isolation that helps do to this figuring out.


Up till now, Problem Isolation has used just performance data measured by our agentless probes (from a product called SiteScope) in order to correlate between a top-line performance metric (like online check-in times) and the health of services that top-line metric depends on (database, app server, integration bus, etc). But there is another source of data we haven't included until now -- the events collected by our operations product, HP Operations Manager. If you have HP Operations Manager, you have a massive source of information that can also be used to determine where top-down performance problems lie. 


 


This is how Problem Isolation now uses HP Operations Manager data:


 



  1. A business service problem is identified. For example, thru synthetic or real-user monitoring we determine that online check-ins are running too slowly

  2. A “time buffer” around the problem start time is determined

  3. The model for the business service in the CMDB is traversed, returning a list of all services supporting the business service

  4. Events that occurred within the above-mentioned time-buffer relating to those supporting services are determined

  5. The services with the best-correlating events (taking into account severity as well) are identified as likely suspects

 


This algorithm applies to any event, whether it’s from a third party enterprise management system (e.g. Tivoli), from HP Operations Manager, or,  from HP Network Node Manager.


 


-------------


 


In our quest to move from service health monitoring to service health management, we're trying to provide as much information  relating to a problem/incident as possible - all in one place in such a way that the information is easily visualizable.


 


In BAC 8.0, you can see the following regarding a problem service, all from one place:


 



  • The current performance of the service

  • The performance of the service over time

  • SLAs resting on the service and their closeness to jeopardy

  • Business processes using that service and the impact of the problem on those business processes. If you have our Business Transaction Management modules of BAC, you can see exactly which business process instances are affected or at risk. In industries like financial services this matters because the value of transactions can vary hugely, and business operations wants to know which important business instances are affected (e.g. A $10m inter-bank transfer) so that they can initiate manual work-arounds

  • Measured user experiences resting on this service. Imagine an app server is having a problem. You can "look upwards" and see that this app server is used to serve the online check-in business service. You can then see the measured impact of the app server problem on the online check-in user experience. This would be measured using either synthetic or real-user monitoring

  • Real changes that have occurred under the problem service. The real changes are inferred by the discovery technology that notices deltas between the state of CIs today versus yesterday

  • Planned changes against the problem service as taken from the change/release management system

  • Outstanding incidents against the problem service. You can "look across" to the details of the incidents to see if they provide the app support team with any insight into how to solve the problem

  • Non-compliancy state of servers under the problem service. Our server automation technology now updates server compliance state into the CMDB and this can be viewed in this 360 degree view of the problem service

 


------------


Mike Shaw.

The new Operations Manager i (OMi)

Fifteen years ago when HP Operations Manager (or OpenView Operations as it used to be called) was released, event management was really “infrastructure event management”. The concepts of middleware, of customer experience, of SOA, and of automated business process didn’t exist. But now they do, and we need a consolidated management solution that does full “consolidated business service management” rather than simply “consolidated infrastructure management” so that all events can come into one place where the operators are highly empowered to deal with quickly and accurately.


 


This is the aim of the Operations Manager i (OMi) product we announced at Vienna Universe on December 9th.


 


OMi and a shared service dependency model


OMi shares the same discovered service model as BAC. The service dependency model holds information on business transactions, customer experience, applications, middleware, infrastructure and now, network information discovered by our network management product, NNMi.


 


Using a common service dependency model means you can look upwards in the model (if the event comes below) and understand what services, user experiences and transactions are affected. The SLAs are in the model - so you can see how they are affected, and how close to jeopardy this problem brings you.


 


The model also tells you changes that have been made under a CI; what changes are planned; and what incidents are outstanding in Service Manager. Also the HP Server Automation product puts the compliance state of servers into the same model, so you can see if anything under a CI is out of compliance.


 


OMi  and root event analysis using a discovered model
The perennial problem with any event management system, whether it be infrastructure or network management, is event noise. A problem with one object can cause a whole array of dependent objects to fire off events too.  For example:


 



  • the SAN has a problem and fires off an event ...

  • the Active Directory using that SAN fires off an event...

  • the Exchange Server using that Active Directory has just lost its directory and fires off an event ...

  • and the proxy server that is driving the web UI to Exchange fires off an event too.

 


Four events  - but one “root event” – one “actionable condition”.


 


Typically, event management systems have created “event correlation languages” so you can program up rules to eliminate these noise events, but such rules are simply not robust to change (and we all know how fast IT systems change). Also, the number of events that can be generated is so large that it’s impossible to write all the rules you need.


 


What we do with OMi is use the service dependency map to get to the root event when a series of events are generated. The actual technology used is the causal engine we released as part of NNMi, but it's using our discovered service dependency map rather than NNMi's discovered network topology.


 


OMi's health views


With OMi you can create health KPIs against the things (CIs) that you are managing. These can be combinations of attributes --  like CPU utilization and free memory on a server so that you can see at a glance the health of the Cis under your management, rather than having to achieve the same thing through wading thru a ton of events.  In other words, OMi is mapping the events onto Cis and building up a health picture for you.


 


OMi and existing HP Operations Manager installations


OMi actually sits on top of existing HP Operations Manager installations. It acts like a manager of mangers for them. It can also take events from other event management systems like SCOM.

One brand new product and two major enhancements to the BSM stack - Vienna HP Software Universe 2008

Today is the first day of our software user conference here in sunny Vienna, Austria. We just announced a brand new product, and two major upgrades.


I'll start with the new product ...


 


HP Operations Manager i (Part of HP Operations Center) is our next-generation consolidated event and performance management product following on from HP Operations Manager. Internally, we call it OMi. Three keys about OMi...


 



  • You can take events from anywhere into OMi because it sits directly on top of our CMDB which holds business transaction, user experience, application, middleware, and infrastructure information.

  • OMi does root event analysis using the discovered service dependency map held in the CMDB. This means that only root events are shown in the console and subsequent events caused by the root event are hidden.

  • OMi gives you more than simply an "event stream" view of the world. It can also give you a service health view of the services you are responsible for. The exact make-up of a service's health is up to you - it will obviously include availability and performance, but it can also include the number of open incidents, for example.

 


I'll write more about OMi in a post at the end of this week.


 


HP Business Availability Center 8.0  (BAC 8.0) for application management uses HP Labs patented statistical analysis to cut through the volume of performance and operations event data in order to help customers predict and proactively resolve business service performance problems before they impact end users.


 


I did a post recently on the difference between user experience monitoring and user experience management noting how important performance problem isolation was. The latest version of BAC 8.0 does such analysis using both performance information and the rich source of events that HP Operations Manager, our operations management software, can give you.


 


I'll post on BAC 8.0 in more detail next week.



HP Network Node Manager i Advanced : we released a brand new network management product, NNMi,  this time last year incorporating a clever root event analysis engine (now found in OM i) and new, much faster spiral network discovery engine. The new Advanced Edition of NNMi is targeted at large enterprises.


 


NNMi Advanced helps you predict the service impact of network degradation before business services are negatively effected through its integration with our CMDB.


 


And it natively uses our run-book automation technology, Operations Orchestration, to automatically collect data, fix problems and verify state once a fix has been actioned.


 


More on NNMi Advanced in the NNM blog. 

From User Experience Monitoring to User Experience Management

This post discusses the differences between monitoring user experience and actually managing user experience to a certain level.  The difference between the two lies mainly in our ability to figure out where user experience problems lie.


 


A recent EMA study found that 52% of user experience problems (problems with things like front-ends to web sites) were reported by customers. In other words, the majority of  user experience problems were found by customers and not by IT.  Not good - this is probably a statistic we might want to keep hidden from the business!


 


How do we get to a situation where we're not using our customers as very expensive monitoring devices, particularly in these days when each piece of  customer business is important?


 


We use "user experience monitoring" or EUM. There are two flavors or EUM - synthetic and real-user.



     


  1. Synthetic monitoring uses scripts to simulate customer activity. Let's imagine we want to ensure our online check-in web user interface performs well at all time. After all, if it doesn't, there is a good chance that customers will not choose our airline again. We record a script that retrieves a dummy booking, chooses a seat, and opts to print a boarding pass. We run this three step script every ten minutes from three different locations around the world. We can now proactively and automatically know when customers are having problems with our online check-in. And because we are monitoring from three different points, we can determine if it's the actual business service or something on the way to the business service that is causing any problems.
     

  2. Real user monitoring is like a network probe. It listens to http network traffic (or tcp/ip traffic) and builds up a picture of the performance of a web user interaction. We would use it to monitor our online check-in process. Real user monitoring gives us a very rich source of diagnostic information should there be a problem.

  3.  


Which should we use, synthetic or real-user monitoring? Both. Synthetic gives you the proactive notification - we have one customer who runs 30 different scripts at 7:30am every morning before its customers come on line so it can proactively fix problems before the customers notice. But real user monitoring gives a richer source of diagnostic information.


 


And that brings me to the title of this blog posting .. "from user experience monitoring to user experience management".


 


Everything I've described right now is all about telling you you have a problem before your irate customers ring in and tell you first. This is important, but it's not the whole story. Imagine I could tell you your car was having problems before you noticed.  If I told you of a problem, your first question would most certainly be, "and what's causing the problem?" 


 


And that's a very important question to ask. Forrester estimates that 80% of the time spent fixing a problem lies in finding where the problem is.  And the typical performance problem of the sort we're talking about ping-pongs between different expert groups as they try to determine whose fault it really is. Going back to your car, it would go to the electrics group first. They would say, "not us". Then to the transmission group. "Not us". And then to the fuel system group. "Not us". Then the wheels and tyres group. "OK - it's us. You need new tyres and a re-alignment".


 


And why does this allocation ping-pong occur? Because we don't give the appropriate tools to first-line support - to the Operations Bridge. What they need is a tool that allows them to figure what is causing a user experience performance problem.


 


We have such a tool (which probably doesn't surprise you - I probably wouldn't write about a problem we most certainly couldn't solve and had no plans to solve!) And this tool moves us from User Experience Monitoring to User Experience Management - to detecting a problem early and then fixing it quickly and efficiently. This tool does the following:



     


  1. It takes you step by step thru a workflow looking at different potential sources of problem cause. Once I've talked about the areas the tool considers, you'll see why a guided workflow is a helpful thing to have.
     

  2. It re-runs any synthetic scripts against the problem business service to see if the problem is still occurring
     

  3. It looks for patterns in the behaviour of the problem business service's performance. Does the problem occur weekly? Is it only from one location?
     

  4. It looks to see if a dependent service is causing the problem. These critical business services are complex beasts - they can depend on a lot of "stuff" underneath - and we always under-estimate just how much "stuff" there is under a business service. A colleague of mine was recently at a customer site. They were estimating how many IT artefacts were under their claims processing system. The consensus was "about 12 systems". They looked in the CMDB and found it was 42 systems, all the way down to network paths, DNS servers and the like.  Anyway - our tool looks at the performance of all these dependent services and uses some clever statistical analysis tools to determine the most likely suspect - the dependent service that most highly correlates with the business service performance problem.
     

  5. It looks to see what changes have occurred under the problem business service.  It gets this information from the discovered CMDB - discovery notices when a service has changed and flags this. There is an IDC statistic that says that if a problem has occurred under a problem service, there is a probably of 80% that the change will have caused the problem.
     

  6. It looks at incidents against this problem service in the service desk. This allows us to get the service desk's view of what is happening.

  7.  


These guided analyses allow us to get a much better idea of what is causing a user experience performance problem, thus allowing us to "cut into the 80%" - cutting into the most time-consuming part of solving a complex performance problem and stopping that inefficient "allocation ping-pong".


 

Why do we need Consolidated Event Management?

A lot of customers we talk to are trying to get to a situation where all events go to one place for initial processing - something often referred to as "Consolidated Event Management". Why this drive - what bad things happen if we don't have Consolidated Event Management?


In broad terms, there are three problem situations:



  • The first is where events do all come to one place (the "operations bridge" or "NOC" or "centralized first level support" - we'll use the term "Operations Bridge" from now on) but the Operations Bridge doesn't have the tools to effectively deal with these events.  Typically there are two ways in which the Operations Bridge is unempowered:

  •  



    • There are so many events that they can't figure out what's going on. They can't see how events relate to each other, so they can't clean away the "event noise" that is being generated.

     



    • There is a performance problem with a service (e.g. Online check-in for an airline). An event comes to the Operations Bridge, but they don't have the tools to figure out what is causing the problem.  Diagnosing such complex performance problems is hard. Firstly because of the sheer amounts of data involved and secondly because the interrelationships between IT elements are complex and numerous.  However, it's important we do furnish the Operations Bridge with tools to deal with these types of problems because otherwise we get "allocation ping-pong" as the problem bounces around the different domain expert groups.

     


  • Second is where all infrastructure events come into one place, but events from monitoring user experience or business transactions go elsewhere. Typically when people say "we are doing consolidated operations" they really mean, "we are doing consolidated infrastructure operations" - there are two parallel systems, one for infrastructure events and one for events from monitoring user experience and business transaction.

  •  


    When we have such a parallel system it's hard to triage user experience and business transaction problems unless you can see all events. You need the events from the infrastructure to understand what's causing user experience / business transaction problems. If you don't, solving such problems takes a long time and results in allocation ping-pong, thus wasting the time of valuable domain experts.


     


  • And the third problem is  that we have a number of different event management systems. Typically there is one for each domain we are managing. This situation is probably the worst because:

  •  




    • Duplication of effort. Events aren't raised in isolation. One system has a problem, which causes a dependent system to have a problem, which causes a dependent system to have a problem, and so on. If each domain is receiving events in isolation, then we have duplication of effort as each domain works on "their problem". In fact, only one domain has a problem - but we can't see that because all events don't come into one place. Domain experts are expensive resources. And if they aren't dealing with problems they can be actually doings that move the business forward rather than simply "keeping the lights on".



    • No overall view of what's happening. When there is a problem with a key business service, the first question the business answered is, "when will it be fixed?"  If all events are going to individual domain consoles, we can't answer that question.  While we can't always give a fix time when all events come to one place, we at least have a good chance of doing so.



    • It's very time consuming and inefficient to solve complex performance problems.  Let's imagine we have an "application performance problem". The event comes straight to the application support guys. They look at it and realize it's not their problem. In other words, the application domain experts have had their time wasted. Had the event gone to a central place, the performance problem could have been correlated against the performance of the dependent infrastructure services, against any events coming out of dependent services, against any changes that have occurred, against historical incident data for that service, and against the compliance state of dependent services.  It would have been allocated to the correct domain group more quickly and by less expensive 1st level support staff.

     


So, if we have events all come to central Operations Bridge where we have the tools to cut out event noise, to triage complex performance problems, and to understand how all IT services relate to each other, we can solve problems faster and in a more efficient way, making good use of our 1st level support and our expensive domain experts.

How does BSM improve IT Operations' efficiency?

Welcome to the BSM BLOG.


I was on the phone last night talking to a two Gartner consultants about a couple of announcements we'll be making soon. Every time I mentioned a new feature, they said, "how does this improve IT operations' efficiency?" And of course they were right to ask - in these hard times, everyone's boss is asking the same thing.


So .. how can BSM help improve IT operations' efficiency?


BSM is all about managing the health of services in a way such that IT's priorities are aligned with the business's priorities - it might almost be better if we called it "Business Service Health Management".


The best thing we can do is ensure that all the key business services are always healthy - ensure that they never have a problem. This may seem like a statement of the obvious, but I think it's an important goal for us BSM vendors to bear in mind. If we can do anything to help pro actively avoid service health problems ever occurring, then that's best. We've started down this path with the proactive anomaly detection technology we introduced in our Problem Isolation product - but we want to take it further in the future.


However - let's imagine we didn't prevent the problem with the health of a service, how can BSM help improve efficiency now? There are a number of problems with solving health problems:




  • Allocation churn: we find out the on line check-in business service for which we are responsible is not performing well. Where does the problem lie? Such a complex business service can rest on 20 or 30 IT artifacts. Which one is the cause? Which team shall we give the problem to? Tell you what - "it looks like a network problem" (i.e. "I've not really got any idea, but my intuition tells me it's the network guys"), let's give it to them. We all know that such complex performance problems "bounce around support" because we don't have the tools to let us analyze exactly where the problem really is. In fact, Forrester estimates that 80% of the solution time for a performance problem is spent figuring out where the problem lies, and only 20% is spent fixing the problem once we know where it lies. So - if BSM can give us the tools to figure out where the problem lies, this will help "cut the 80%".



  • "Domain expert inefficiency": In IT support, we typically have our first line support, and behind them, second and third level support. We often refer to the 2nd and 3nd level support groups as "domain experts". And when domain experts are not fixing support problems, they can be "moving the business forward" - merging duplicate IT systems resulting from an acquisition, moving more infrastructure to virtualized systems, etc. In fact, IDC estimates that 73% of IT budgets are spent "keeping the lights on". If we could be more efficient in our use of domain experts, we could shift some of this 73% towards things that give us consolidated billing systems, consolidated ordering systems, consolidated HR systems, etc. So, how do we make our domain experts more efficient?

    Let's imagine all events, from everything from business transactions, thru user experience problems, applications, middleware down to infrastructure and networks all come to one place. And let's imagine that the first level support is actually empowered to understand each one of these event types - they have tools that clean out irrelevant events, understand the business priority of each event, execute automated run-books to fix simple problems, and figure out who to give complex problems to without causing churn. We could then use our domain experts more effectively.  The experts wouldn't get events that hadn't been pre-processed by 1st level support. They would get incidents that were caused by their domain - no allocation churn. And all "trivial stuff" would have been filtered out allowing them to focus on just those incidents which 1st line really couldn't fix.

For the BSM BLOG team - Mike Shaw.

Search
About the Author(s)
  • Anil is an enterprise software professional with 15+ years of experience. He has both breadth and depth of understanding in IT Infrastructure management including Network, System, Storage, Virtualization and Cloud. As a product manager, Anil had successfully introduced many new products into the world wide market.He innovates on regular basis and he holds many patents.
  • Doug is a subject matter expert for network and system performance management. With an engineering career spanning 25 years at HP, Doug has worked in R&D, support, and technical marketing positions, and is an ambassador for quality and the customer interest.
  • Drew is a subject matter expert for the BSM product structure, the BSM simplification program and the BSM Customer Appreciation Program. With a career spanning 10 years at HP, Drew has worked in Consulting and Product Management on various HP Software management products.
  • Dan is a subject matter expert for BSM now working in a Technical Product Marketing role. Dan began his career in R&D as a devloper, and team manger. He most recently came from the team that created and delivered engaging technical training to HP pre-sales and Partners on BSM products/solutions. Dan is the co-inventor of 6 patents.
  • This account is for guest bloggers. The blog post will identify the blogger.
  • Manoj Mohanan is a Software Engineer working in the HP OMi Management Packs team. Apart being a developer he also dons the role of an enabler, working with HP Software pre-sales and support teams providing technical assistance with OMi Management Packs. He has experience of more than 8 years in this product line.
  • Architect and User Experience expert with more than 10 years of experience in designing complex applications for all platforms. Currently in Operations Analytics - Big data and Analytics for IT organisations. Follow me on twitter @nuritps
  • Pranesh Ramachandran is a Software Engineer working in HP Software’s System Management & Virtualization Monitoring products’ team. He has experience of more than 7 years in this product line.
  • Ramkumar Devanathan (twitter: @rdevanathan) works in the IOM-Customer Assist Team (CAT) providing technical assistance to HP Software pre-sales and support teams with Operations Management products including vPV, SHO, VISPI. He has experience of more than 12 years in this product line, working in various roles ranging from developer to product architect.
  • Ron is a subject matter expert for BSM\APM, Currently in the Demo Solutions Group. Ron have over thirteen years of technology experience, and a proven track record in providing exceptional customer service. He began his career in R&D as a software engineer, and team manager.
  • Stefan Bergstein is chief architect for HP’s Operations Management & Systems Monitoring products, which are part HP’s business service management solution. His special research interests include virtualization, cloud and software as a service.
Follow Us
Twitter Stream


HP Blog

HP Software Solutions Blog

Labels
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation