OVO UXMONbroker cause high load and hang servers( solaris 10 containers) (329 Views)
Reply
Regular Visitor
unixkl
Posts: 1
Registered: ‎09-29-2013
Message 1 of 2 (329 Views)

OVO UXMONbroker cause high load and hang servers( solaris 10 containers)

[ Edited ]

Hello,

 

Does anyone know about such issue where UXMONbroker generate a high load on servers and hang.

servers are solaris containers, 1 Global zone and 2 local zones, all 3 become very slow response time and affect customer application. it make difficult to investigate when issue is happening

 

Only workaround is to kill all process and stop OVO.

this issue appear for the third time this year

 

From global zone:

root# ps -ef |grep -i UXMONb*

    root 22793 22468   1 16:42:04 ?           4:59 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x ss
    root 23405  1116   1 16:48:41 ?           0:16 /usr/bin/sh /var/opt/OV/bin/instrumentation/UXMONswapmon -l /var/opt/OV/log/OpC
    root 22922 22266   1 16:43:28 ?           3:27 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x ac
    root 22927 22468   1 16:43:31 ?           3:59 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x rc
    root 23449 22580   0 16:49:03 pts/14      0:04 grep -i UXMONb*
    root 22923 22267   1 16:43:28 ?           4:09 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x ac
    root 23071  1011   1 16:45:04 ?           2:06 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x sv
    root 22944  1011   1 16:43:38 ?           3:10 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x rc
    root 23218  1011   1 16:46:36 ?           1:09 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x cr
    root 22891  1011   1 16:43:04 ?           3:33 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x kt
    root 22794  1011   1 16:42:05 ?           4:12 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x ss
    root 23211 22468   1 16:46:31 ?           1:41 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x cr
    root 23068 22468   1 16:45:03 ?           2:38 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x sv
    root 22889 22468   1 16:43:02 ?           4:22 /opt/OV/nonOV/perl/a/bin/perl /var/opt/OV/bin/instrumentation/UXMONbroker -x kt
root#

 

root OpC # prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  1727 8398124  4787M 4503M cpu10   49    0 287:55:47 2.3% java/652
 22636 root     3608K 3496K cpu32   58    0   0:03:37 1.5% prstat/1
 22931 root     2992K 2304K cpu40   60    0   0:00:52 1.4% iostat/1
 22190 root     7136K 6032K cpu3    60    0   0:07:32 1.3% perl/1
 22140 root     5808K 4768K cpu2    60    0   0:07:06 1.2% perl/1
 22793 root     3480K 2512K cpu6    60    0   0:01:24 1.2% UXMONbroker/1
 22247 root     7072K 5976K cpu28   60    0   0:06:39 1.1% perl/1
 22927 root     1520K 1488K cpu2    60    0   0:00:35 1.1% ksh/1
 22938 root     2640K 1968K cpu14   60    0   0:00:38 1.1% vmstat/1
 22880 root     8128K 5216K run     60    0   0:00:47 1.1% sendmail/1
 22794 root     3416K 2232K cpu16   60    0   0:01:22 1.1% UXMONbroker/1
 22923 root     3416K 2240K cpu5    60    0   0:00:40 1.1% UXMONbroker/1
 22870 root     8328K 3568K run     60    0   0:01:01 1.1% sendmail/1
 22470 root       25M   15M cpu11   60    0   2:00:14 1.1% opcle/2
 24984 8398126  3780M 3621M run     53    0 267:51:43 1.1% java/537
 22188 root     5552K 4504K cpu19   60    0   0:06:15 1.0% perl/1
 22930 root     8264K 3440K run     60    0   0:00:38 1.0% sendmail/1
 22944 root       23M 1888K run     60    0   0:00:34 1.0% opcacta/1
 22468 root       23M   13M sleep   59    0   1:35:10 1.0% opcacta/4
 22889 root     3416K 2192K cpu12   60    0   0:00:55 1.0% UXMONbroker/1
  1116 root       25M   15M cpu24   60    0   2:03:59 0.9% opcle/2
 22922 root     3352K 1744K cpu28   60    0   0:00:30 0.9% UXMONbroker/1
 22939 root     1520K 1488K sleep   59    0   0:00:29 0.9% ksh/1
  1011 root       23M   14M sleep   59    0   1:32:15 0.8% opcacta/4
  2232 8398127  2671M 2458M cpu4    59    0  70:02:39 0.8% java/544
  5867 8398123  4714M 3741M cpu7    29    0  68:32:11 0.7% java/401
 22840 root     8328K 3264K run     60    0   0:01:00 0.6% sendmail/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
   233 root      628M  662M   1.0%  30:06:10  29%
     2 8398124  4746M 4515M   6.9% 287:55:47 2.3%
     2 8398126  3743M 3640M   5.6% 267:51:43 1.1%
     2 8398127  2630M 2473M   3.8%  70:02:39 0.8%
     2 8398123  4677M 3750M   5.7%  68:32:11 0.7%
     2 8398125  3622M 3286M   5.0%  29:59:52 0.6%
     2 10237026 2087M 1995M   3.0% 172:07:46 0.1%
     5 smmsp    6008K   12M   0.0%   0:14:14 0.1%
Total: 307 processes, 5950 lwps, load averages: 65.39, 57.25, 35.70

 

 

Changing the running intervals of df_mon.cfg and many other *.cfg file in /var/opt/OV/conf/OpC still dont help much

 

root OpC # grep -i interval df_mon.cfg
interval = 20

 

Please help! anyone face this already?

 

cheers

 

 

P.S. This thread has been movedfrom ITRC HP Systems Insight Manager Forum to Systems Management (OpenView-OP Mgmt) Support and News Forum. -HP Forum Moderator

Valued Contributor
Nestor Herrera
Posts: 142
Registered: ‎06-13-2011
Message 2 of 2 (213 Views)

Re: OVO UXMONbroker cause high load and hang servers( solaris 10 containers)

I have the same problem, have you found any solution?

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.