Loadrunner 11.52 "generator using over 80% CPU" (6993 Views)
Reply
Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 1 of 33 (6,993 Views)

Loadrunner 11.52 "generator using over 80% CPU"

[ Edited ]

Load generator CPU usage issue:  1 controller, 4 generators; now all running Loadrunner version 11.52, on Windows server 2008 R2 SP1 virtual machines.  For 100-, 300-, 500- virtual user tests, which we traditionally have run on our customer-facing web site prior to releasing updates to production, we now constantly get "generator using over 80% CPU" errors every time scenarios run.  This did not occur prior to upgrading from 11.0 to 11.52; or moving from Windows 2003 to Windows 2008 R2.  Other than being VM's, all machines are well within HP recommended specs for memory, CPU, paging, etc.  However, Windows PerfMon bears out the error message, when executed on the generators during load tests:  the generators constantly get pegged at 100% CPU for practically the whole execution of the scenario.  We've even seen this "generator >80% CPU" error with as few as FIVE (5) virtual users; though certainly more prevalent with larger number of simultaneous users.  We run virtual users as threads; this is to HTTP/HTML app, not a new app, never had this issue previously.  See attachment for more info / screen captures.  Any advice?  That is, other than loading up generators with multiple and/or multi-core CPU's?  <--Or, is that the only solution??  Anyone else seen this CPU issue on load generators, with either LR 11.52, or with VM generators on Windows server 2008 R2?  Note:  load gen servers each have multiple IP addresses assigned to them, and performance tests all invoke "IP Spoofing."  Has no one else experienced this particular "CPU >80%" issue with load generators?  Thank you.  Need a solution ASAP.

HP Expert
Suerte13cr
Posts: 736
Registered: ‎09-14-2011
Message 2 of 33 (6,962 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

This could be several things, but using 2008 machines on VMs is common.  If you are using web vusers should be a small footprint.

 

-Do you have admin rights on the system.

-Is CPU consumption above 80% during the entirety of the test

-Any mismatch in versions in patch version

 

Regards,

Mario M.

HP LR/PC Support

If you are customer you can try posting on our internal forums for more personalized support.

LoadRunner:
http://h30499.www3.hp.com/t5/LoadRunner-Support-Customer/bd-p/loadrunner-support-customer-forum

Performance Center:
http://h30499.www3.hp.com/t5/Performance-Center-Support/bd-p/performance-center-support-customer

How to:
http://www.youtube.com/watch?v=4ChlysWupWM

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 3 of 33 (6,964 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

-Admin rights:  yes.  I take it that by this you mean the service(s) on controller and/or generators for Loadrunner having admin rights:  that's a yes.  If you mean otherwise, please let me know.

 

-CPU consumption above 80% for entirety of the test, with few exceptions:  usually at full 100% CPU.

 

-No mismatch in LR version patches; everything at LR 11.52:  controller & generators.  As far as I know, Windows 2008 has same patches on all servers; same for .NET.  Are you indicating that Windows and/or .NET patch discrepancies would cause this high CPU usages on generators?  (I assume it would be discrepancy between controller <--> generator windows patches)

 

Unfortunately, we have "official" support for HP Loadrunner through SAP:  it's how the contract is written.  I'd hoped someone else had seen this same situation, knew an easy fix, or something to try to alleviate the situation.

Occasional Visitor
INC_2013
Posts: 1
Registered: ‎08-22-2013
Message 4 of 33 (6,955 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi,

 

We have been experienced the above issue with LR 11.50 during our Load Testing. please advice..

 

Thank you,

-Irfan

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 5 of 33 (6,924 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Even DOUBLING the CPU's on generator server makes little difference-- really need help on this-- & solution. Has no one else had this issue, found root cause, & applied a fix for it?
HP Expert
Editus
Posts: 650
Registered: ‎04-06-2011
Message 6 of 33 (6,921 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hello

For me it looks like permissions related, try to run the agent using "Run as administrator" and run it as process or simply try installing a LG on a NON server windows, such as windows 7 o windows 8, this in order to verify or deny the permissions are related, same for controller, if it is running on a server windows version, then use "run as administrator" and avoid using the controller as LG as well.

 

let us know the outcome

 

regards

Frequent Advisor
KF1
Posts: 59
Registered: ‎11-30-2011
Message 7 of 33 (6,915 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

I already have everything installed and am having the same issue.  Hits 80% wayyyyy too quickly and then sits ~100% for whole test, with very few users.

 

All Windows server 2008 R2 SP1 and Oracle 11.2.0.3 DB.

 

ALM 11.52 server

PC 11.52 server

PC-host 11.52 (C+DP)

PC-host 11.52 (LG)

PC-host 11.52 (LG)

 

Where can I configure this step below??

 

try to run the agent using "Run as administrator"

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 8 of 33 (6,909 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Same question as kf:  how do I 'run the agent using "Run as administrator"'

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 9 of 33 (6,896 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

I double-checked that we're unning Load Generator Agent Service on all 4 generator servers with login that has admin rights on those servers. So don't think permissions are the problem.
Frequent Advisor
KF1
Posts: 59
Registered: ‎11-30-2011
Message 10 of 33 (6,894 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

do me a favor grey and let me know what you did to confirm that if you would... i installed using a local admin account, but I can only assume they run as IUSR_METRO which is admin.

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 11 of 33 (6,889 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

I "cheated:" I asked our admin-server guy, who set up the Windows 2008 servers, and our logins; other than that, I execute All Programs>HP Software>Tools>Agent Runtime Setup Configuration, "Allow Virtual Users to Run on this Machine without Login," entering my login domain, name, & password. I know my login has admin rights on the generator servers: I can execute other server manager functions, as well as restart the servers, with that login. That's why I don't think it's an "access to the server" permissions issue for the Loadrunner agent service.
Frequent Advisor
KF1
Posts: 59
Registered: ‎11-30-2011
Message 12 of 33 (6,886 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Yeah my setup has the first radio button selected also... the password field is blank currently (assuming thats how it is for security purposes)

 

Did you install the 11.52 Patch 1 for your install?  I am wondering if that could help, but have not installed it yet for ALM/PC yet

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 13 of 33 (6,889 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

I have not installed any patches except items delivered originally in 11.52 service pack:  1 for controller, 1 for load generators, 1 for analysis, & 1 for vugen.  I'll look into HP for additional 11.52 patch.  I've just about concluded that 11.52 is completely incompatible with Windows Server 2008 R2, our standard virtual server configuration.  I have no confidence whatsoever in any load tests we've executed since installing 11.52:  I think all are compromised by high CPU usage on load-gens.  I cannot recommend that we stay with HP Loadrunner if we cannot get this fixed ASAP.  I absolutely will be recommending another performance testing package, if solution for LR is not forthcoming.

HP Expert
Editus
Posts: 650
Registered: ‎04-06-2011
Message 14 of 33 (6,877 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi

You go to the start up menu, lelect the app icon, do right click and there is the option "runs as administrator", on this post is about LoadRunner, so IUSR_Metro is out of context, that is for PC only and there are no issues related to permissions on PC, but we know in LoadRunner there are, so please do the following

 

In your computer or in any other running a non server version of windows, please install a Loadgenerator and verify if using only this LG you can reproduce the issue, the idea is isolate the problem and focus on the root cause, so the firs is to confirm if this is related or not to the windows server permissions, if you are able to reproduce the same behavior on a non server LG then we will focus on the scripts.

 

regards

HP Expert
_Buzzy
Posts: 105
Registered: ‎08-10-2011
Message 15 of 33 (6,842 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi Everyone,

 

I am Boris from the LR R&D team.

 

We are currently investigating this issue. It would help us greatly if you could send me some run results from the controller when this happened. If possible please log the CPU usage of all the processes on the LG or attach a screenshot of the process that causes the 100% cpu spike.

 

Also note that there was a change in the functionality of the "80% CPU" warning in the controller between 11.04 and 11.52, in 11.52 we made it much more sensitive so it is possible you are still running above 80% CPU in 11.04 and simply don't get the warning about it.

 

Please email me to kozo[a]hp.com

 

Thanks,

Boris.

Frequent Advisor
KF1
Posts: 59
Registered: ‎11-30-2011
Message 16 of 33 (6,838 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Boris,

 

We are still in our 'setup' phase of our HP ALM/PC setup and I have not had the time/resources to fully test.  I did get this message with like 3 users running a login script which scares me.

 

Again I have not done full testing, but when I do, should I still be sending you my results?  Or do you only deal with LR and not PC?  Thanks.

HP Expert
_Buzzy
Posts: 105
Registered: ‎08-10-2011
Message 17 of 33 (6,813 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi,

 

Sorry for the belated reply, I saw your message just now.

I would appriciate anything you can send. We are investigating the issue and need all the input we can get.

 

Thanks,

Boris.

Valued Contributor
Deb_1
Posts: 136
Registered: ‎09-01-2010
Message 18 of 33 (6,810 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Guys,

 

had upgraded to Lr 11.52 few months back since then it is not same as old good LoadRunner (till 11.0). I have been getting this issue more frequently on top of Vugen slowness. On top of this that abnormal termination of mmdrv process is a nightmare.

Currently I am working with SAP & HP guys to fix this issue (predominantly my testing focus around SAP and Web)

Can see HP have added few things (like link to community center, software auto update) which I assume not required.

 

I think this release of LoadRunner 11.52 is the first release with many added feature like correlation studio, VTS but this is not a stable release.

 

If you are happy with current version of 11.04 then stick to it until HP comes with more stable version. 

Thanks
Deb
HP Expert
_Buzzy
Posts: 105
Registered: ‎08-10-2011
Message 19 of 33 (6,738 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi,

 

Please check if you have the following components installed on your LG machine:

1) McAfee Antivirus

2) Symantec NetBackup

 

If you have both installed, please consult this thread for a possible solution.

 

Furthermore, for McAfee Antivirus please disable On-Access scan of the entire system or at least for the LoadRunner processes and specifically mdrv.exe and mmdrv.exe. For Symantec NetBackup you can try installing this patch which fixes the high CPU utilization.

 

 

Note that you get the "over 80% CPU" error even if the CPU is being used by a non LoadRunner process. The LG monitors the total CPU usage windows counter and reports based on that. Also note that this error was added in the latest releases of LR so running the same script with LR 11.04 will never yield this warning even if the CPU utilization is 100% for the entire load test.

 

Please tell me if this solution help/didn't help you.

 

Thanks,

Boris.

 

 

HP Expert
Editus
Posts: 650
Registered: ‎04-06-2011
Message 20 of 33 (6,721 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi

I am curious, you know about this only because of the error messages right?

but how is this actually behaving on both version, on 11.52 you probably will see the CPU use on task manager above the 80% but what about 11.0? the fact that you dont get the errors doesnt mean it is behaving better, can you please confirm this.

 

 

regards

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 21 of 33 (6,705 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Don't really have option of uninstalling 11.52 & re-installing 11.0, in order to make the comparison.  While it's true that we only started looking at this this after getting the "80% usage" message upon installing & 11.52, we found that the load generators WERE using 100% of available CPU during our most frequent HTML web-based tests.  And, knowing (and reading & researching) that generators utilizing over 80% of their own CPU cannot be good for a load test, we're looking for solution.  Latest we're reviewing is suggestion from your expert Boris:  to see if either anti-virus or network backup software is interfering with, and/or adding to CPU usaging for, load test processes/threads on generator servers; don't have answer on that yet.

Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 22 of 33 (6,698 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

From my operations/network folks: "We are not running Netbackup nor McAfee on these servers."
Advisor
greyhelm
Posts: 13
Registered: ‎08-09-2013
Message 23 of 33 (6,678 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

After having HP guys view our controller & generator servers in operation during load testing, I have come to the following conclusions:

1.  The ">80 CPU" issue probably existed prior to our upgrade to LR 11.52 & prior to upgrading our servers to Windows 2008 R2.  I have no proof of this; but am about 89% satisfied that our generators were overloaded, for the same scenarios,  before these upgrades.

2.  The "real" solution will likely involve upgrading our load generator virtual machines, increasing their CPU capacity and/or memory, and same on the controller server; plus, maybe even increasing  number of load generators we utilize (currently, we use four).

3.  We might also need to revisit manner in which we test some of our apps, to assure scenarios give good analytic results without overloading generators' CPUs.  It appears  ">80%" error occurs when  scenario slams generator(s) with more virtual users than can be  processed simultaneously.

4.  In future, at times, it may prove valuable to include at least one LG's stats, for CPU, etc., in output, so this gets included in analysis for specific load tests-- to assure LG's aren't overloaded, in resulting report (added via System Resource Graphs, Windows Resources, on Run screen in Controller).

***Thanks for Boris & team for assisting on this!

Occasional Visitor
garylim
Posts: 1
Registered: ‎09-22-2013
Message 24 of 33 (6,625 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

[ Edited ]

Hi,

 

We are encounter the same issue of 100% CPU load for only 10 virtual users running on Core Duo 2.6GHz, 4 GB RAM physical laptop.

 

1. We are running LR 11.52 for Controller and Generator

 

2. We are performing HTTP test scenario for 5 static URL using 10 vuser with ramp rate of 2 vuser per 15sec 

 

3. We are using the following Setting:

- Logging Enable (Standard log, Send only when error occurs)

- Run Vuser as a thread

- Define each action as a transaction

- Limit Think Time to 2 sec

- Pacing (as soon as previous iteration ends

 

4. Any other setting we need to consider to optimise the load operation?

 

Please advise.

Occasional Visitor
nikil1043
Posts: 1
Registered: ‎09-26-2013
Message 25 of 33 (6,592 Views)

Re: Loadrunner 11.52 "generator using over 80% CPU"

Hi Boris,

 

With LR 11.5 and with very less number of virtual users we are facing the same issue of CPU usage exceeded 80%.We have never seen this message with the previous versions.Please suggest.

 

 

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.