08-22-2013 07:10 AM - edited 08-22-2013 12:38 PM
Load generator CPU usage issue: 1 controller, 4 generators; now all running Loadrunner version 11.52, on Windows server 2008 R2 SP1 virtual machines. For 100-, 300-, 500- virtual user tests, which we traditionally have run on our customer-facing web site prior to releasing updates to production, we now constantly get "generator using over 80% CPU" errors every time scenarios run. This did not occur prior to upgrading from 11.0 to 11.52; or moving from Windows 2003 to Windows 2008 R2. Other than being VM's, all machines are well within HP recommended specs for memory, CPU, paging, etc. However, Windows PerfMon bears out the error message, when executed on the generators during load tests: the generators constantly get pegged at 100% CPU for practically the whole execution of the scenario. We've even seen this "generator >80% CPU" error with as few as FIVE (5) virtual users; though certainly more prevalent with larger number of simultaneous users. We run virtual users as threads; this is to HTTP/HTML app, not a new app, never had this issue previously. See attachment for more info / screen captures. Any advice? That is, other than loading up generators with multiple and/or multi-core CPU's? <--Or, is that the only solution?? Anyone else seen this CPU issue on load generators, with either LR 11.52, or with VM generators on Windows server 2008 R2? Note: load gen servers each have multiple IP addresses assigned to them, and performance tests all invoke "IP Spoofing." Has no one else experienced this particular "CPU >80%" issue with load generators? Thank you. Need a solution ASAP.
08-22-2013 07:50 AM
This could be several things, but using 2008 machines on VMs is common. If you are using web vusers should be a small footprint.
-Do you have admin rights on the system.
-Is CPU consumption above 80% during the entirety of the test
-Any mismatch in versions in patch version
If you are customer you can try posting on our internal forums for more personalized support.
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
08-22-2013 08:59 AM
-Admin rights: yes. I take it that by this you mean the service(s) on controller and/or generators for Loadrunner having admin rights: that's a yes. If you mean otherwise, please let me know.
-CPU consumption above 80% for entirety of the test, with few exceptions: usually at full 100% CPU.
-No mismatch in LR version patches; everything at LR 11.52: controller & generators. As far as I know, Windows 2008 has same patches on all servers; same for .NET. Are you indicating that Windows and/or .NET patch discrepancies would cause this high CPU usages on generators? (I assume it would be discrepancy between controller <--> generator windows patches)
Unfortunately, we have "official" support for HP Loadrunner through SAP: it's how the contract is written. I'd hoped someone else had seen this same situation, knew an easy fix, or something to try to alleviate the situation.
08-26-2013 01:20 PM
08-26-2013 09:32 PM
For me it looks like permissions related, try to run the agent using "Run as administrator" and run it as process or simply try installing a LG on a NON server windows, such as windows 7 o windows 8, this in order to verify or deny the permissions are related, same for controller, if it is running on a server windows version, then use "run as administrator" and avoid using the controller as LG as well.
let us know the outcome
08-27-2013 10:22 AM
I already have everything installed and am having the same issue. Hits 80% wayyyyy too quickly and then sits ~100% for whole test, with very few users.
All Windows server 2008 R2 SP1 and Oracle 188.8.131.52 DB.
ALM 11.52 server
PC 11.52 server
PC-host 11.52 (C+DP)
PC-host 11.52 (LG)
PC-host 11.52 (LG)
Where can I configure this step below??
try to run the agent using "Run as administrator"
08-28-2013 10:27 AM
08-28-2013 10:29 AM
do me a favor grey and let me know what you did to confirm that if you would... i installed using a local admin account, but I can only assume they run as IUSR_METRO which is admin.
08-28-2013 11:42 AM
08-28-2013 11:58 AM
Yeah my setup has the first radio button selected also... the password field is blank currently (assuming thats how it is for security purposes)
Did you install the 11.52 Patch 1 for your install? I am wondering if that could help, but have not installed it yet for ALM/PC yet
08-28-2013 01:25 PM
I have not installed any patches except items delivered originally in 11.52 service pack: 1 for controller, 1 for load generators, 1 for analysis, & 1 for vugen. I'll look into HP for additional 11.52 patch. I've just about concluded that 11.52 is completely incompatible with Windows Server 2008 R2, our standard virtual server configuration. I have no confidence whatsoever in any load tests we've executed since installing 11.52: I think all are compromised by high CPU usage on load-gens. I cannot recommend that we stay with HP Loadrunner if we cannot get this fixed ASAP. I absolutely will be recommending another performance testing package, if solution for LR is not forthcoming.
08-28-2013 02:34 PM
You go to the start up menu, lelect the app icon, do right click and there is the option "runs as administrator", on this post is about LoadRunner, so IUSR_Metro is out of context, that is for PC only and there are no issues related to permissions on PC, but we know in LoadRunner there are, so please do the following
In your computer or in any other running a non server version of windows, please install a Loadgenerator and verify if using only this LG you can reproduce the issue, the idea is isolate the problem and focus on the root cause, so the firs is to confirm if this is related or not to the windows server permissions, if you are able to reproduce the same behavior on a non server LG then we will focus on the scripts.
08-29-2013 08:25 AM
I am Boris from the LR R&D team.
We are currently investigating this issue. It would help us greatly if you could send me some run results from the controller when this happened. If possible please log the CPU usage of all the processes on the LG or attach a screenshot of the process that causes the 100% cpu spike.
Also note that there was a change in the functionality of the "80% CPU" warning in the controller between 11.04 and 11.52, in 11.52 we made it much more sensitive so it is possible you are still running above 80% CPU in 11.04 and simply don't get the warning about it.
Please email me to kozo[a]hp.com
08-29-2013 08:35 AM
We are still in our 'setup' phase of our HP ALM/PC setup and I have not had the time/resources to fully test. I did get this message with like 3 users running a login script which scares me.
Again I have not done full testing, but when I do, should I still be sending you my results? Or do you only deal with LR and not PC? Thanks.
08-31-2013 11:37 PM
Sorry for the belated reply, I saw your message just now.
I would appriciate anything you can send. We are investigating the issue and need all the input we can get.
09-01-2013 01:04 AM
had upgraded to Lr 11.52 few months back since then it is not same as old good LoadRunner (till 11.0). I have been getting this issue more frequently on top of Vugen slowness. On top of this that abnormal termination of mmdrv process is a nightmare.
Currently I am working with SAP & HP guys to fix this issue (predominantly my testing focus around SAP and Web)
Can see HP have added few things (like link to community center, software auto update) which I assume not required.
I think this release of LoadRunner 11.52 is the first release with many added feature like correlation studio, VTS but this is not a stable release.
If you are happy with current version of 11.04 then stick to it until HP comes with more stable version.
09-10-2013 02:59 AM
Please check if you have the following components installed on your LG machine:
1) McAfee Antivirus
2) Symantec NetBackup
If you have both installed, please consult this thread for a possible solution.
Furthermore, for McAfee Antivirus please disable On-Access scan of the entire system or at least for the LoadRunner processes and specifically mdrv.exe and mmdrv.exe. For Symantec NetBackup you can try installing this patch which fixes the high CPU utilization.
Note that you get the "over 80% CPU" error even if the CPU is being used by a non LoadRunner process. The LG monitors the total CPU usage windows counter and reports based on that. Also note that this error was added in the latest releases of LR so running the same script with LR 11.04 will never yield this warning even if the CPU utilization is 100% for the entire load test.
Please tell me if this solution help/didn't help you.
09-11-2013 02:09 PM
I am curious, you know about this only because of the error messages right?
but how is this actually behaving on both version, on 11.52 you probably will see the CPU use on task manager above the 80% but what about 11.0? the fact that you dont get the errors doesnt mean it is behaving better, can you please confirm this.
09-12-2013 10:52 AM
Don't really have option of uninstalling 11.52 & re-installing 11.0, in order to make the comparison. While it's true that we only started looking at this this after getting the "80% usage" message upon installing & 11.52, we found that the load generators WERE using 100% of available CPU during our most frequent HTML web-based tests. And, knowing (and reading & researching) that generators utilizing over 80% of their own CPU cannot be good for a load test, we're looking for solution. Latest we're reviewing is suggestion from your expert Boris: to see if either anti-virus or network backup software is interfering with, and/or adding to CPU usaging for, load test processes/threads on generator servers; don't have answer on that yet.
09-17-2013 06:27 AM
After having HP guys view our controller & generator servers in operation during load testing, I have come to the following conclusions:
1. The ">80 CPU" issue probably existed prior to our upgrade to LR 11.52 & prior to upgrading our servers to Windows 2008 R2. I have no proof of this; but am about 89% satisfied that our generators were overloaded, for the same scenarios, before these upgrades.
2. The "real" solution will likely involve upgrading our load generator virtual machines, increasing their CPU capacity and/or memory, and same on the controller server; plus, maybe even increasing number of load generators we utilize (currently, we use four).
3. We might also need to revisit manner in which we test some of our apps, to assure scenarios give good analytic results without overloading generators' CPUs. It appears ">80%" error occurs when scenario slams generator(s) with more virtual users than can be processed simultaneously.
4. In future, at times, it may prove valuable to include at least one LG's stats, for CPU, etc., in output, so this gets included in analysis for specific load tests-- to assure LG's aren't overloaded, in resulting report (added via System Resource Graphs, Windows Resources, on Run screen in Controller).
***Thanks for Boris & team for assisting on this!
09-22-2013 09:26 PM - edited 09-22-2013 09:41 PM
We are encounter the same issue of 100% CPU load for only 10 virtual users running on Core Duo 2.6GHz, 4 GB RAM physical laptop.
1. We are running LR 11.52 for Controller and Generator
2. We are performing HTTP test scenario for 5 static URL using 10 vuser with ramp rate of 2 vuser per 15sec
3. We are using the following Setting:
- Logging Enable (Standard log, Send only when error occurs)
- Run Vuser as a thread
- Define each action as a transaction
- Limit Think Time to 2 sec
- Pacing (as soon as previous iteration ends
4. Any other setting we need to consider to optimise the load operation?
09-26-2013 08:07 AM
With LR 11.5 and with very less number of virtual users we are facing the same issue of CPU usage exceeded 80%.We have never seen this message with the previous versions.Please suggest.