Running a HP OMi proof-of-concept and other lessons from the road

I have just returned from my global voyage to the United States from India. I spent two weeks visiting with customers and experiencing the coldest weather I think I have ever survived.  I was meeting with customers, along with an implementation partner, to run a proof-of-concept of our OMi solutions stack. 

 

Some technical tips

 

1. OMi graphs not presenting data - how to troubleshoot

 

Is your OMi Performance perspective not working? Are you unable to draw graphs and charts? Need a quick way to troubleshoot the issue?

Try the following -

 

java -jar <ovinstalldir>/java/jcodautil.jar –ping –n <hostname.domain>
java -jar <ovinstalldir>/java/jcodautil.jar –obj –n <hostname.domain>

NOTE: If java is not in the path on the BSM system, you can use the java runtime under <TOPAZ_HOME>/jre/bin.

The first command above does a connectivity check with the collection agent on the target system <hostname.domain>

The second command retrieves objects for which data collection has happened on the target system <hostname.domain> 

 

2. vPV collection, java heap size

 

Have you recently setup HP vPV and are seeing collection failures with vCenter? It might be a problem of lack of memory with the data collections done by vPV's java-based vCenter collector. You can confirm this by looking into the /var/opt/OV/log/status.virtserver logfile on the vPV system for errors such as the one below.

 

ERROR [2014-02-12 09:36:03,621] : Thread[Thread-19,10,main] Exception details are : java.lang.OutOfMemoryError: GC overhead limit exceeded

 

To fix this, you need to increase the heap space maximum size occupied by the java runtime. This is a setting in the file /var/opt/perf/viserver.properties.

 

Try changing  "jvmArgs=-Xms256m –Xmx1536m –classpath ..."  to  "jvmArgs=-Xms256m –Xmx3072m –classpath ...". Restart the pvcd service (ovc -restart pvcd) once you are done with this change. Ensure that the java process with the above arguments is also recycled (ps -eaf  | grep java | grep Xmx). If the java process listing is not showing the new arguments then kill it and restart pvcd again.

 

NOTE: If you have close to or more than 1000 VMs in your data center, you might want to increase the vPV memory allocation to 8gb (default allocation is 4 gb) along with the heap space setting above.

 

3. Content Pack updates from HP Live Network for HP BSM 9.2x

If you are installing BSM 9.22 or 9.23 (or any future versions) you need to keep in mind that the content pack updates for Infrastructure monitoring are no longer shipped with the OMi server updates. This has to be downloaded from HP Live Network and updated/installed separately. FYI, you would also find Oracle, Hadoop and Vertica content packs (and corresponding management packs for OMi-MA) listed in this location.

 

4. Non-root agents - setting bbc_port to connect to on OM and BSM server

If OM agents are running as non-privileged user (NPU, or non-root) they are most probably configured to communicate on a non-standard port (not our usual 383 port).

 

# managed node setting
[bbc.cb]
SERVER_PORT=2006

It is important the the OM server (as well as the OMi server) are aware of this change. To make this change, use the bbc.cb.ports::PORTS setting.

# management server setting
[bbc.cb.ports]
PORTS=node1:13001,node2:443,node3:12345

You could also provide patterns of ip addresses and host names in the PORTS setting. Use 'bbcutil -getcbport <nodename>', to check if  port settings are configured correctly.

 

5. Solaris zones - OM agent running as non-root user

Have you tested running the OM agent 11.13 on a solaris 11 global zone as a non-privileged user even though it is not officially supported. This works fine and we were able to get data for the non-global zones also (in the BYLS metric class). Note here that since the OM agent is not running as 'root' user there is a limitation that the agent will not obtain the process counters for any user other than the agent user itself since sufficient access rights are not present.

 

6. IBM HMC support, frame CPU utilizationIBMHMC_TOPOLOGY_VISPI-2.png

With the 11.12 agent upwards there is support for obtaining data from the IBM HMC (Hardware management console) that manages the p-series frame (the IBM physical box running on Power architecture, hosting many logical partitions - LPARs). With this, you can know exactly how much CPU the frame is using, which helps a lot in the capacity planning angle and knowing how much latent power is left unused on the hardware. With the latest Infrastructure Content Pack updates, we also map these elements in the BSM Run-time service model. See example model on the right.

 

7. DNS resolution, reverse lookup

In far too many occasions we noticed some major functionality in the setup failing mainly due to hostname resolution failing or incorrect reverse lookups. It is important to ensure prior to running a proof-of-concept that DNS setup is sane. If there's no DNS, entries must be adequately added in the '/etc/hosts' files.

 

8. Monitoring BSM software systems -

Some of the crucial HP BSM software pieces must be monitored - their health must be kept up at all times to make sure these monitoring systems keep working. We are building a system health monitoring piece as an add-on to SHR. This will send out alerts to HP Operations Manager when there is degraded performance in SHR.

 

9. Working with code snippets -

There were code snippets and command line examples that I need to copy and run on the customer system. Rather than email out the stuff to a gmail or yahoo account, I used github gists for this - and this became my virtual clipboard on the cloud. I can also create a secret gist that no one else can access. Try out github - it is free, pretty cool stuff.

 

 

Non technical stuff

 

1. Wool gloves and winter shoes

If you are going to be in icy chill (sub-zero) and there's winds blowing, you got to be wearing leather gloves not wool ones. The wind also seeps right through wool socks - these don't offer much comfort either - don't plan on standing too long outdoors.

If you are on business travel, remember to carry two pairs of shoes: One simply used for walking on snow - these can be your sneakers (or tennis shoes). And the other is your formal shoe which you use within the office. [Apparently all that salt on the ground can harm your leather shoes].

 

2. Frozen Niagara falls

After 120 years the extreme winter caused the niagara falls to freeze up. Here's a pic from twitter.

 

3. Grand Canyon

When flying over the Grand Canyon area from Charlotte to Sunnyvale, the flight steward announced 'To your left you see the Grand Canyon'. 25 minutes later I saw that we had still not crossed the Grand Canyon area. I realized how vast the park really is, while flying at 33000+ feet.

 

4. California roads

The state of the roads in the bay area is bad. The ride on these roads is really bumpy - wonder where all the IT company tax dollars go. Don't even get me started on Lombard St.!

 

5. Vegetarian sushi (at London Heathrow aiport)

 Yes there's something as veg sushi. It tasted great, yum!

Photos of Itsu, London
This photo of Itsu is courtesy of TripAdvisor

Comments
| ‎02-14-2014 07:00 AM

Thanks for the info, great stuff.

SanjayChaudhary | ‎02-17-2014 09:05 AM

Nice article and equally attractive pictures .

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
Ramkumar Devanathan (twitter: @rdevanathan) works in the IOM-Customer Assist Team (CAT) providing technical assistance to HP Software pre-sa...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation