OM-OMi: Dealing with Event Duplicates, effectively…

Should you be concerned about event duplicates in your Operations Manager (OM) or Operations Manager i (OMi) setup? What harm they could do, you may ask. Well, if you have a large environment with thousands of events coming into OM and OMi on a daily basis,  large duplicate counts can have some performance implications (believe me some customers have events with duplicate count running in 3 to 5 digits). But you don't need to worry about it. There are enough configuration options in Operations Management to handle this situation.

 

Why Event Duplicates?

  •  Duplicate events are event reporting the same or similar information about something
  • The same event being sent repeatedly so it is shown as one event with increasing message/event count (called duplicate count) in the event browser on the server
  • Sometimes (or more often?) poor monitoring and event management practices cause loads of duplicates

Duplicates in HP Operations Manager (OM Unix/Linux/Windows)

In the HP OM(U/L/W) world, two events are considered duplicates when either they have the same “key” attribute OR in case of messages with no keys, a set of event attributes are exactly the same. In such a case, the duplicate count of the original event is incremented while the second event itself may be dropped or stored in the database as per the configuration.

 

In OMW, you can configure many aspects related to duplicates -

omw-settings.PNG

 

Some important configuration settings define:

omw-settings-2.PNG

  •  Which attributes of the event should match for them to be duplicates?
  •  Should duplicate messages be stored in the data base? If yes, how many? (You can also set it to unlimited.)
  •  Should the severity and message text of the original messages be shown or that of the latest duplicate message?
  •  Should annotations be added for duplicate messages and if so, how many?
  •  In flexible management environment, what all event changes to forward and what all to accept to/from other management servers

omw-settings-3.PNG

 

For more details about message suppression configuration, refer to OMW online help, section - “configure duplicate message suppression”

 

In OMU/L, some of the similar configuration options for duplicates are:

  • Attributes used to identify duplicates are fixed and cannot be modified
  • Enable/disable duplicate suppression and addition of duplicate as annotation to the original message using following command on the server:

                 opcsrvconfig -dms -enable anno

  • Most of the configuration is done by setting server configuration variables using a command like:

                 ovconfchg -ovrg <OV_resource_group> -ns opc –set <variable name> <variable value>

  • Some important configuration variables are:

                OPC_MAX_DUPL_ANNO: maximum no of annotations for duplicates

                OPC_SUPPRESS_DUPL_MSG_KEY_ONLY: suppress duplicates based on message keys i.e. attributes based suppression is not done if set to TRUE

               OPC_UPDATE_DUPLICATED_SEVERITY: update the severity of the original message with that of the duplicate message

               OPC_UPDATE_DUPLICATED_MSGTEXT: update the message text of the original message with that of the duplicate message

               OPC_SEND_MSG_COUNT – enable/disable sending of duplicate count in manager to manager forwarding

               OPC_ACCEPT_MSG_COUNT – enable/disable receiving of duplicate count in manager to manager forwarding

 

For more details on message suppression in OMU/L, refer to OM Concepts Guide and OM Server Configuration Variables documents.

 

Duplicates in HP Operations Manager i (OMi)

Similar to OM, OMi also provides duplicate suppression with somewhat different configuration options.

 

OMi-duplicate-settings-2.PNG

 

One important difference is that OMi does not store the duplicate messages in the database. Instead it drops them completely after updating the original event. Some configuration options are:

  •  Duplicates based on keys (just like in OM) – enabled by default
  •  Duplicates based on ETIs(same CI, same HI and same HI Value) – enabled by default
  • Duplicates based on attributes (like in OM) – disabled by default
  • Should the title and severity of the original message be updated from the newer duplicate message?
  • Should history entries be added for duplicate count changes on the event? (Please note that this infra setting to disable history entries for duplicates is applicable only for events that are de-duplicated by OMi.)

For more details, refer to OMi Application Admin Guide and look for section “duplicate events suppression settings”

 

Duplicates in the synchronized OM - OMi Environment

When OM is integrated with OMi to forward all events, new events as well as event updates from OM are forwarded into OMi. The event updates include change in duplicate count, message title, message severity, lifecycle state, custom message attributes and annotations. These synchronized updates have certain implications on OMi which are important to understand:

  • Every event update coming from OM into OMi, adds a history entry to the event. So, if there are hundreds of updates on an event, then there will be those many history entries under the history tab of the event details window in the OMi event browser.
  • When OM performs de-duplication, it can result in the  addition of multiple history lines (for duplicate count, for annotations, for title and severity change , if applicable and depending on the settings in OM) to the corresponding OMi event. Please do not forget that configuration setting in OMi to “not generate history lines for duplicate messages” applies only when OMi de-duplicates the events and not in this case.

omi-history-entries-2.PNG

 

As a result of above, for a single event in event database table, there could of hundreds of records in the tables for storing history lines and event changes. Such events with large number of history lines can cause significant performance impact on the OMi event browser and event archival, as has been seen with many large customers. There could be other factors impacting the performance, this is just one of the factors.

 

So, what should be done to avoid these performance issues?

 

Firstly, consider the reasons for so many duplicates being generated in the monitoring setup. If possible, modify your monitoring to reduce/avoid the duplicates.

 

If you cannot change the monitoring and do not care about the duplicate counts in OMi, then

  • If events are coming into OMi from non OM sources and OMi is doing de-duplication, then by default , history lines will not be added as per the infrastructure setting mentioned in earlier section; make sure this setting is set to false
  • If events are coming from OM into OMi and OM is doing the de-duplication, configure OM to not send the ‘change in duplicate count’ update to OMi by setting “operations to be forwarded” configuration in OMW; (in OMU/L, you can use OPC_SEND_* variables to enable and disable the similar settings) ;
  • Even when sending of the duplicate count change is disabled, if a duplicate suppression adds annotation entries in OM and/or changes event title, severity, those updates will still be sent to OMi and add history entries unless you have configured otherwise. It will be good to disable adding of annotation entries for duplicates in OM using relevant configuration settings.
  • You also have the option to disable the duplicate suppression in OM and instead do it in OMi. This allows OMi to make sure no history entries are added for duplicate count changes. But this is not a recommended practice, especially in large environments with huge number of incoming events. In the end-to-end event flow, data should be suppressed as early as possible to avoid unnecessary network traffic and processing on higher level servers.

 

These are just some simple and quick ways to configure event duplicates to avoid potential performance issues caused by large number of history lines and event changes in OMi.

 

Stay tuned for more detailed document on this topic but in the meanwhile feel free to share your comments/feedback/questions….

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.