Re: lsof cpu/memory utilization is high (1533 Views)
Reply
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 1 of 13 (1,582 Views)
Accepted Solution

lsof cpu/memory utilization is high

HI Expert,

 

I have issue with  CPU and memory Utilization is high and some process occured due to this. when i checked into the server i noticed that rds service is higher.

 

My objective is

 

to findout what is root cause why cpu/memory utilization is high and this server running Data Protector.

 

I have some question

 

1.  did maxfiles related to this issue? so i need to extend more value on maxfiles_lim? or i need to put new request to upgrade memory? see below

 

server2b# lsof |wc -l
4284
server2b# kctune |grep -i maxfiles_lim
maxfiles_lim                       4096  Default     Immed


2. when i execute below command i found some process error, are this affect to cpu/mem utilization?

server2b#lsof -c rds

.

.

.

rds     27873 root  106u  unix         0x2497b980        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8247d100)
rds     27873 root  107u  unix         0xdb292680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xc007b100)
rds     27873 root  109u  unix         0xda92e700        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xe7e3400)
rds     27873 root  110u  unix          0xa4c0d00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x819da380)
rds     27873 root  111u  unix         0xcc634380        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd2c62380)
rds     27873 root  117u  unix         0xaad47680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xa8a64a00)
rds     27873 root  120u  unix          0x8435680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xcc079100)
rds     27873 root  121u  unix         0xbfa3fd00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8443700)
rds     27873 root  123u  unix         0xa9970a00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xf933fd00)
rds     27873 root  125u  unix         0xdb292c80        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd87f1080)
^C

 

3. Total file running under rds is 145, are this normal?

server2b#lsof -c rds |wc -l
145


4. Top out showing user is high load

 

CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
 0    1.39  53.5%   0.0%  37.2%   9.3%   0.0%   0.0%   0.0%   0.0%
 2    1.44  73.0%   0.0%  20.7%   6.4%   0.0%   0.0%   0.0%   0.0%
 4    1.40  60.2%   0.0%  31.8%   8.0%   0.0%   0.0%   0.0%   0.0%
 6    1.43  57.7%   0.0%  33.4%   8.9%   0.0%   0.0%   0.0%   0.0%
---   ----  -----  -----  -----  -----  -----  -----  -----  -----
avg   1.42  60.9%   0.0%  30.8%   8.3%   0.0%   0.0%   0.0%   0.0%

 

Please help me on this

 

Thanks

 

BR

Naj


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Acclaimed Contributor
Dennis Handly
Posts: 24,383
Registered: ‎03-06-2006
Message 2 of 13 (1,576 Views)

Re: lsof cpu/memory utilization is high

>1.  Is maxfiles related to this issue? so I need to extend more value on maxfiles_lim?

 

maxfiles_lim is a policeman.  It prevents processes from going crazy.  But some programs need to have larger values.  The sysadmin needs to judge whether a useful application is more important that this rogue processes.

 

I'm not sure I see the connection between the default lsof output and maxfiles_lim?

 

>2. when I execute below command I found some process error, are this affect to cpu/mem utilization?

 

This is typically sloppiness in the code.  It may be trying to tell you that there is something wrong and may be just a warning.

 

>4. Top out showing user is high load

 

It looks like a lot of time is spent in the kernel.  Are the data protector processes the top ones?

 

Why are you using lsof to debug this performance problem?  Any particular reasons?

Do you have glance installed?  There is a trial version.

 

Please use plain text.
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 3 of 13 (1,571 Views)

Re: lsof cpu/memory utilization is high

HI Handy,

 

> Are the data protector processes the top ones?

Yes, rds are the top ones

 

>Why are you using lsof to debug this performance problem?  Any particular reasons?

Do you have glance installed?  There is a trial version.

I've glance installed in server, i used that but it just giving basic info. Im looking more details, what the process running on that time causes processing high and is this because of any error trying to attempt. From that, i'll be able to further troubleshot

 

I've little knowledge in data protector, but if you guys have some info please share.

 

1. How to manage DP process in HPUX, i mean to reduce rds CPU processing in one time. if CPU/memory high load, it would be affect to backup activity and many process would be occur.

 

Thanks

 

BR

Naj


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 4 of 13 (1,566 Views)

Re: lsof cpu/memory utilization is high

Any updates?


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Acclaimed Contributor
Dennis Handly
Posts: 24,383
Registered: ‎03-06-2006
Message 5 of 13 (1,558 Views)

Re: lsof cpu/memory utilization is high

>I've glance installed in server, I used that but it just giving basic info.

 

glance gives you all the details you would ever want.  I don't see lsof helping here.

 

>what the process running on that time causes processing high

 

glance will give that.

 

>I mean to reduce rds CPU processing in one time. If CPU/memory high load, it would be affect to backup activity and many process would be occur.

 

I assume you want high CPU processing by rds so you can get faster backups.

Please use plain text.
Outstanding Contributor
Pete Randall
Posts: 16,201
Registered: ‎11-03-1996
Message 6 of 13 (1,554 Views)

Re: lsof cpu/memory utilization is high

If Omniback/DataProtector is interfering with production then you either need to schedule your backups for off hours or move your OB/DP cell manager to another box.


Pete
Please use plain text.
Honored Contributor
Hakki Aydin Ucar
Posts: 1,302
Registered: ‎08-16-2006
Message 7 of 13 (1,544 Views)

Re: lsof cpu/memory utilization is high

when DP is running , in most case the "rds" utilization is normal. this issue is new OR happens now and then ?

did you try a restart omni;

 

/opt/omni/sbin/omnisv -stop

/opt/omni/sbin/omnisv -start

 

sometimes it works.

Please use plain text.
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 8 of 13 (1,542 Views)

Re: lsof cpu/memory utilization is high

HI

 

Below is current reading on glance

 

 Glance C.04.70.001             20:54:34+     ia64                                                  Current  Avg  High
-----------------------------------------------------------------------------------------------------------------------------------
CPU  Util   S                     SAU                                          U                                  | 68%   68%   68%
Disk Util   F FV   V                                                                                              |  7%    7%    7%
Mem  Util   S                         SU        U                                                                 | 85%   85%   85%
Swap Util   U                   UR           R                                                                    | 33%   33%   33%
-----------------------------------------------------------------------------------------------------------------------------------
                                                           PROCESS LIST                                                Users=    3
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   ( 400% max) Cnt  IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
rds                27873 root        143.9    31   0.0   434.1mb  466.5mb SLEEP
glance             28714 root         18.2     1  29.2     696kb    940kb SLEEP
crs                27895 root          8.1     1   0.0    16.8mb   18.0mb SLEEP
ksh                19508 root          6.1     1   0.0     612kb    656kb  PIPE
ovconfd             2510 root          1.4    11   0.0     8.9mb   19.9mb SLEEP
midaemon           12128 root          0.7     4   0.0    41.1mb   42.5mb SLEEP
bsm                10039 root          0.0     1   0.0    35.2mb   36.6mb SLEEP
cmcld              28072 root          0.0    10   0.0    33.3mb   33.4mb SLEEP
coda               12132 root          0.0     9   0.0   596.3mb  661.3mb SLEEP
vxfsd                 70 root          0.0    15   8.5    24.4mb   27.4mb SLEEP
scopeux            12092 root          0.0     1   0.0    28.0mb   33.8mb SLEEP
httpd               9308 hpsmh         0.0    27   0.0    36.2mb   54.4mb SLEEP
kcmond              2667 root          0.0     1   0.0    41.3mb   41.8mb SLEEP
ovcd                2212 root          0.0    28   0.0    21.2mb   48.1mb SLEEP
cimprovagt          1866 root          0.0    26   0.0   145.6mb  188.1mb SLEEP
cimprovagt          1867 root          0.0     4   0.0    24.2mb   35.6mb SLEEP
bsm                10090 root          0.0     1   0.0    24.5mb   26.4mb SOCKT


Based on glance, rds consume 143.9 % CPU and current load for CPU is 68%, are this normal? sometime it hit till 80% constantly. 

 

Highlight

 


-This server running on cluster, any possible cluster has any affect (in sense of processing) to DP activity?

- When i execute lsof, i found died path/error path, are this will be affected to CPU load, my logic is, DP trying to trigger the path for sometime and this will be hang up and consume CPU load, are this true?

- If DP need to be tuning back, what the item need to emphasis ?

 

 

Why i ask that, because same configuration implement on other cluster server and their DP processing looks more stable rather than this server. So i would suspected, it must be *something* that must investigate. Since, i don't have knowledge in Data Protector, so  i need your kind advice on this to improve CPU load to be more stable.

 

Thanks

 

BR

Naj


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 9 of 13 (1,541 Views)

Re: lsof cpu/memory utilization is high

Hi Hakki,

Restarting is only works for temporary, CPU load back to high after 2/3 hours.


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Outstanding Contributor
Pete Randall
Posts: 16,201
Registered: ‎11-03-1996
Message 10 of 13 (1,540 Views)

Re: lsof cpu/memory utilization is high

Well, you could always stop running backups.  That will certainly reduce your CPU load. :smileywink:


Pete
Please use plain text.
Honored Contributor
Hakki Aydin Ucar
Posts: 1,302
Registered: ‎08-16-2006
Message 11 of 13 (1,533 Views)

Re: lsof cpu/memory utilization is high

I cannot say normal due to it depends on your system perfomance and capacity , and conditions like backup is running OR another performance issue effect this. But I remember more than 100% utily of rds in some customer. According to your CPU usage no cpu bottleneck yet, but 144% seems a bit high.

it reminds of something I faced in past,  I had solved similar issue with a missing linker patch ,but my hw was PA-RISC not Integrity..

Please use plain text.
Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 12 of 13 (1,529 Views)

Re: lsof cpu/memory utilization is high

Hi Hakki,

>I had solved similar issue with a missing linker patch
could you please more details on this,means after patch the process avg below 100%?


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Please use plain text.
Honored Contributor
Hakki Aydin Ucar
Posts: 1,302
Registered: ‎08-16-2006
Message 13 of 13 (1,524 Views)

Re: lsof cpu/memory utilization is high

>could you please more details on this,means after patch the process avg below 100%?
All I remember about similar issue (that is not same because my omni start got stuck with high rds not started and reached above 100%), but I saw %90 around in most of healthy systems. So , somehow 100% not indicates a problem directly as I said before. 

My solution was to put patch PHSS_41214 , watch out for this PA-RISC only !

I hope it igives some clue and my explanation makes sense.

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation