Who’s up for some mobile performance testing?

Everyone seems to be asking about performance testing in relation to the new reality of mobile access to almost every publicly available application, so I’m picking up the gauntlet and answering the tough questions...

 

How are mobile devices changing the way customers work?

A mobile device can be a smartphone, tablet, mini-tablet, phablet, or other small handheld computer.  Gartner says that by 2013 (yes, now!), there will be more mobile devices in use than traditional computers. This rise in the number of mobile devices is changing the way people use applications, and has the following implications:

  • Organizations must make their applications accessible to mobile users
  • Users expect a different experience when using their mobile device than with a PC:
    • The UI needs to be customized to fit the mobile device’s screen
    • The user expects fewer clicks to achieve their goal
    • Apps should respond quickly.  If they don’t, users are likely to abandon them.
  • For the best experience, organizations develop native apps to be downloaded onto the mobile device. This is in addition to the existing browser application, because infrequent users won’t want to download the dedicated app for occasional use. They’ll use their mobile device’s browser, but the application will need to respond differently when accessed by a mobile browser, to account for the different screen size etc.

What challenges do mobile devices present to app providers?

Here are some of the challenges that application providers need to address:

  • Both dedicated apps and browser-based applications need to support the plethora of mobile device types, models, platforms and platform versions
  • In order to provide mobile browser users with a mobile look and feel, applications are often redirected to a different server which serves up pages designed for mobile. Many organizations need to provide and support both native apps and browser-based applications.
  • Mobile devices use different networks (eg. 3G, 4G, LTE, etc.) with different bandwidths, and the application is expected to perform well on each of them. There are also network conditions that are common to mobile applications, such as latency, packet loss and dropped connections.

Yes, but how does that affect performance testing?

Let’s look at the ways performance testing for mobile applications differs from traditional performance testing

  • Mobile devices hold connections longer than regular devices. The implication is that there is a rise in the number of concurrent open connections. Your application, together with its backend servers and infrastructure, needs to cope with this rise to ensure that the connections aren’t exhausted and that other users aren’t denied service.
  • Data consumption from mobile devices is typically slower than regular services.  Applications need to ensure that this will not cause a drain on system resources.
  • Mobile devices communicate with cells, and as the device changes location, the cell it communicates with changes. This can result in a bandwidth change, and even complete drop-outs. This can affect the way the application behaves.
  • To record a script for performance testing, you need to be able to record the activity of users working on mobile devices.  But:
    • You can’t simulate it using a regular browser, because the network traffic sent from a mobile browser is different, both in the content of the UI that is served, and the HTTP headers.
    • Nor can you use a regular browser to simulate a native application hosted on the mobile device.

You can also check out this blog post from John Jeremiah which discusses how usage and deployment of mobile applications differ from that of traditional ones. He discusses how mobile applications raise new challenges, particularly around mobile performance testing.

 

I see.  So how do I record scripts for load testing then?

Performance testing software will take care of most of the hard work for you. HP LoadRunner is the industry-standard software for performance testing. It can test an application under the load of tens, hundreds or even thousands of potential users. HP LoadRunner load tests your application by emulating the environment where multiple users work concurrently. While the application is under load, LoadRunner accurately measures, monitors and analyzes a system’s performance and functionality.

 

How does LoadRunner work?

HP LoadRunner consists of three main components – Virtual User Generator (VuGen), Controller and Analysis.

The VuGen component is an Interactive Development Environment (IDE) that records and tunes real business processes that a user would perform in production. These can then be played back to emulate the real users. Using minimal hardware resources, the Controller component then emulates hundreds or thousands of concurrent virtual users to apply production workloads to almost any application platform or environment. HP LoadRunner stresses an application from end-to-end—applying consistent, measurable and repeatable loads—then uses the data to identify scalability issues that can affect real users in production. 

 

As it drives load against the system, HP LoadRunner captures end-user response times for business processes and transactions.  It determines whether the application can meet the required service level agreements. Non-intrusive, real-time performance monitors obtain and display performance data from every tier, server and system component. It can also collect application-tier and code-level data from HP Diagnostics.

 

After the load test completes, the Analysis component provides a single view of end-user, system-level and code-level performance data. It includes a patented AutoCorrelation engine to scan end-user, system, and diagnostics data to provide the most likely causes of system slowdown. The data is correlated to quickly pinpoint problem areas and identify the root cause of performance bottlenecks. This helps your engineers quickly determine whether they have met their performance goals.  If they have not met these goals it helps them identify the reason and who owns the problem.

 

What support does LoadRunner offer for mobile applications?

HP LoadRunner 11.50 has two new protocols for helping to record mobile applications(*):

  • Mobile Application – HTTP/HTML, for recording scripts at the transport level for both browser-based mobile applications and native mobile applications, that communicate with their servers over HTTP
  • Mobile TruClient, for recording scripts for browser-based mobile applications through the browser-based user interface.  This protocol is based on Ajax TruClient, using a browser modified to emulate the mobile browser.

These protocols are independent of mobile operating systems, so will work on different versions of iOS, Android, Windows Mobile, WebOs (Palm), Blackberry, etc.

 

This table will help you choose the right protocol:

 

Application Type

Mobile TruClient

Mobile Application – HTTP/HTML

Browser-based app

Yes

Yes

Native app

 

Yes

 
 If your mobile app is browser-based, and supports Firefox(**), it’s a no-brainer. With the Mobile TruClient protocol, you select the type of mobile device you want the Firefox browser to emulate, and then perform the actions inside the browser just as if you were performing it on the mobile device.

 

There are some other considerations to take into account when using the Mobile Application-- HTTP/HTML protocol though.  Here’s another table which sums them up:

 

Scenario

If…

Then…

Proxy Recording

- The mobile device can be connected to the same network as the VuGen machine, and;

- The app allows proxy configuration

You can configure your mobile app to use VuGen proxy(***) to record the traffic

Server-Side Recording

- You don’t want (or can’t) record from the actual device, and;

- The device, server, and VuGen machine are in the same network, and;

- You only need to record on one server

Install VuGen’s Mobile Sniffer Agent on the server.  You can then use the protocol’s ‘Record and Analyze Traffic’ feature, which will use the agent to record the traffic

Script from Network Capture

You already have a network capture file, such as WireShark’s ‘pcap’ file

Use the protocol’s ‘Analyze Traffic’ feature to turn it into a script

Device Emulator Recording

- You don’t want (or can’t) record from the actual device, and;

- The mobile OS is Android, and;

You have a device emulator available

Record from the emulator by using the protocol’s ‘Emulator Recording’ feature

On-Device Recording

- Your Android device is rooted (to allow access to the device’s system)

You can install the LoadRunner Mobile Recorder on the device (***), and use it to record the traffic as a standard ‘pcap’ file. You can then open the file on your VuGen machine to create a script.


Where do the network capture files come from, in the ‘Script from Network Capture’ scenario?

Recording application traffic to a capture file is effective when you are unable to record an application using VuGen, as is the case with mobile applications. A capture file is a trace file containing a log of all TCP traffic over the network (although you should note that only HTTP traffic will be included in the script generated from the ‘Analyze Traffic’ option). Using a sniffer application, such as WireShark, you can obtain a dump of all of the network traffic. The sniffer captures all of the events on the network and saves them to a capture file. To generate a smaller, more manageable script, you should try to capture the network traffic only for the time that you perform actions in your application.  To further reduce the script’s complexity, you can configure the ‘Analyze Traffic’ option to filter the data by the client’s or server’s IP.

 

Depending on your operating system and device, you have many options on where and how to record a capture file. See the ‘Recording Traffic into a Capture (Sniffer) File’ section of the LoadRunner documentation for more details.


Right.  You mentioned network speeds?

Different cellular networks have different speed settings, which can have an effect on the application. You should make sure the app can cope with a variety of networks. LoadRunner lets you simulate different network bandwidths for a large number of standard networks. If that’s not enough, you can configure your own values if you’re expecting unusual speeds from network providers in specific geographies.

 

You should think about how different groups of users will be affected by their networks, and configure your scenarios accordingly in LoadRunner’s Controller.

 
And what about things that can go wrong with the network?

You mean packet loss, latency, and things like that?  Yes, there are lots of network issues. You should know that delays in a mobile application will cause users to abandon the application.  The longer the delay, the more users you’re going to lose—and some of these users are never going to use your application again. Some of these delays might be a result of the application itself, but delays are also a ‘feature’ of the communications infrastructure that the mobile device itself is using.  Here are some examples:

  • Limited bandwidth – mobile networks typically have a smaller bandwidth than wireless networks, which means it takes data longer to get through
  • Network latency – the mobile device needs to communicate with a cellular tower, which in turn needs to send the data off to the server, and the response comes back through the tower to the mobile device.  All this takes time.
  • Packet loss – data packets being transmitted between any two points on the network can get lost, or corrupted. Although the network generally corrects these problems by itself, it will cause an additional delay

 

In order to accurately simulate your application’s behavior under these circumstances, LoadRunner is integrated with Shunra’s Network Virtualization (NV), which provides the following capabilities:

  • Native integration with HP LoadRunner and HP Performance Center—because of the tight integration, no additional scripting requirements are needed
  • Robust network virtualization—each load generator can emulate a different location with a unique set of network conditions. This enables the simulation of multiple user populations, and more accurately reproduces actual conditions that can affect the end-user experience
  • Location-aware analytics—results are integrated with the load test results, which gives location-specific performance information, identification of poorly performing transactions, and root causes of performance issues
  • Automatic optimization recommendations—this is Shunra’s proprietary performance scorecard. It grades application performance and offers suggestions to improve performance.

 

For more details on Shunra NV, check out the datasheets and white paper from http://www.shunra.com/products/shunra-nv-hp-software

  
Is there anything I should know about functional testing of mobile apps?

That’s a great question. Although we’ve been talking about performance testing, we should really throw in a few words about functional testing.

 

Your performance testing should be based around real scenarios that users are likely to perform. You need to ensure that your application behaves correctly in when it’s being used by an end-user. This is what we mean by functional testing.

 

Mobile applications are expected to meet the same quality standards as desktop applications and desktop browser-based applications—whether they are native applications or mobile browser-based applications. This is why functional testing of the mobile application should be integrated into your application lifecycle management infrastructure.

You’ve probably heard of HP Unified Functional Testing (UFT), HP’s flagship functional testing product. HP UFT Mobile extends UFT to enable full end-to-end testing of mobile applications on real devices. UFT Mobile lets you develop and run automated tests on multiple devices. It provides a dedicated cloud-based environment of mobile devices to support your testing needs, and makes them available to you wherever you are. 

 

So together with HP LoadRunner and HP Performance Center, HP UFT and HP UFT Mobile can introduce real device application side testing into your mobile load and performance tests. This full, end-to-end testing allows you to create very realistic test scenarios that will closely match and predict what the end user will actually experience. Check out HP’s mobile testing suite here.

 

Summary

We’ve seen how to record and prepare scripts for representing real users on the mobile devices that your application will run on.  Once you’ve decided which network conditions that you want to emulate, build your load testing scenario in the same way as you’d build it for traditional load testing, and you continue to have the full power of LoadRunner at your fingertips.

 

When you run the load test, you can monitor and measure the metrics you’d normally measure in a load test. For example these metrics include: server load and utilization, database and backend server performance, etc.  And you can see exactly how your emulated network behaves by looking at the relevant network emulation monitors. When the load test has finished running, you have access to all of the data collected during the execution of the load test. You can correlate data, analyze how specific virtual users, or scripts, affected system performance, and how network conditions impact the behavior of the system, and ultimately, the end-user’s experience.

 

This should help you answer some of the questions that mobile applications raise:

  • With so many users using so many different devices and versions of operating systems, how can I validate the performance of my application?
  • How can I be sure my customers can reach the application’s servers?
  • What’s the end-to-end user experience like?
  • How does the addition of a mobile interface affect my existing users?

 

You’ll find more information about LoadRunner and the mobile protocols, as well as a fully functional trial version at http://hp.com/go/loadrunner.

 

If you have any comments on this, or if you have more advice to offer around performance testing of mobile applications, leave a comment below.


 ---------------
(*) Actually, these new protocols have been available in LoadRunner since version 11.00 Patch 3, but have been significantly enhanced in version 11.50 and later.
(**) Which is the case with practically all standard mobile web applications, as they use standard HTML
(***) The VuGen Proxy and the LoadRunner Mobile Recorder are available in version 11.52 and later of LoadRunner

Comments
wilsonmar | ‎02-27-2013 09:18 AM

Excellent summary, Malcolm!

For a geeky dive into LoadRunner bits, see 

http://wilsonmar.com/mobile_vugen.htm#Flowchartz

(on a desktop browser)

| ‎02-27-2013 09:52 AM

Thanks Wilson - good job on the flowchart on your site!

Ravi SUvvari(anon) | ‎03-05-2013 12:02 AM

 

Hi ,Interesting Article,

 

I am also interested in Knowing what is approximate load,how many concurrent users in a typical day it can support(i am not sure what is typical load in a day for providers ),How many Vusers are supported for each Lg(Load injektor).

Ravi

http://www.linkedin.com/pub/ravi-suvvari/1/232/23

‎03-05-2013 04:29 AM - edited ‎03-14-2013 06:46 AM

Hi Ravi,

 

Thanks for commenting.  Unfortunately, I can’t give you a specific number of Vusers per load generator, as each situation is unique.  You’ll need to perform some calculations to get a value for your specific situation.  I consulted with my LoadRunner colleagues to come up with a suggestion for how you might go about it...

 

Firstly, identify a single Vuser footprint as follows:

  • Create a scenario in the Controller which uses your script, and set the number of iterations to 30 in the Run Time Setting.  It should be configured to run until complete.
  • Using the Windows Resource Monitor, configure %Process Time and Private Bytes counters on the mdrv.exe process in order to measure the footprint. Note that in order to configure these counters, you should first run the script to have mdrv.exe running. Run the scenario just for the sake of configuring the counters and then stop and save it before initiating the real test.
  • Run the scenario and make sure the monitor data is collected
  • As soon as the scenario execution ends, analyze the results using the Analysis tool. Make sure you analyze complete data and not only summary data.
  • Open the Windows Resource graph and make a note of the average CPU and peak memory utilization.

 

Now, you can determine the number of Vusers per LG.  You need to work out how many Vusers the CPU can sustain, and how many the memory can accommodate.  

 

  • This formula will give you the number of Vusers the CPU can sustain:

     Number of Vusers per CPU = (70% * Number of Core Processors)/ Vuser Average CPU Usage

                     (We recommend limiting it to 70% so as not to overuse the CPU)

 

  • This formula gives you the number of Vusers the memory can accommodate:

     Number of Vusers by Memory = (Total GB RAM of LG – GB RAM allocated for the operating system and other processes) / Vuser Peak Memory Usage

                     (You might want to reserve 1GB for the OS etc)

 

The number of Vusers that you are looking for should be the lower of these two values.

 

Here’s a table with some examples:

No of Core Processors

Average CPU usage of Vuser (%)

Vusers for CPU

(70%*No of Core Processors)/Average CPU usage of Vuser

Total Assignable Memory (GB)

(Total Memory-1GB)

Peak Memory per Vuser (MB)

Vusers for Memory

(Total Assignable Memory/Peak memory per Vuser)

Lower of “Vusers per CPU” and  “Vusers by Memory”

8

10

(70*8)/10=56

8-1=7

80

7GB/80MB=89

56

4

22

(70*4)/22=12

8-1=7

100

7GB/100MB=70

12

8

6

70*8/6=93

16-1=15

120

15GB/120MB=125

93

 

It’s not bulletproof – different scripts have different considerations and requirements, so it’s difficult to say categorically that this will work.  But it should be enough to get you started.

 

You can also post on the LoadRunner Community discussion board to ask other users to share some numbers with you.

NaveenKumar Namachivayam(anon) | ‎03-05-2013 08:34 AM

Very insight information about number of vusers per LG. Thanks!

| ‎05-08-2013 12:00 AM

Update: I added a line to the Scenarios table in the article for On-Device Recording.  Now that LoadRunner 11.52 has been released, the LoadRunner Mobile Recorder is now available from Google Play, so you can record scripts directly on your mobile device!

Satzz | ‎08-07-2013 09:29 AM

Hi,

 

Won't there be any difference if LG is a computer machine for a mobile based script?

if LG is a computer machine, Will it capture the actual response time as if we use in mobile?

 

Thank you

Sathish

| ‎08-08-2013 12:29 AM

Hi Sathish,

 

It depends on what you mean by “actual response time”. With any of LoadRunner’s mobile solutions, the LG will correctly capture the actual response time at the transport level, as it will be sending the same HTTP requests as a mobile device would send, thus causing the server to respond as if it’s responding to a mobile device.

 

If, however, by “response time” you mean the end-to-end time, including both the communications between client and server and the time it takes for the mobile application to process the response, then your mileage may vary. The closest you will get to this is by using the TruClient Mobile solution.

iGattus(anon) | ‎08-26-2013 02:02 PM

Nice Information , Now that there is a LR Recorder for Android. It will be intresting to know if similar app is available for iOS  & BB . Any inputs to how this is beign done currently withour on device tools to caprute will be awesome

| ‎08-26-2013 11:57 PM

Hi iGattus,

 

We’re working on an iOS solution. Due to various Apple-induced restrictions, creating an iOS application that records the traffic is a much more challenging task, but it is in our roadmap and we’re researching various options. There are no plans for a Blackberry app. Blackberry users can use the rest of our mobile recording solutions (proxy, server-side etc.)

naresh_perf | ‎12-03-2013 11:23 PM

Hi MalcolmIsaacs,
Thanks for the table above providing your view on point around consideration of Vusers per load generator..
Can you also throw some light on licencing model of LoadRunner? Can you provide info about factors on which the licensing depends for mobile app performance testing (both native and web based apps)? Which protocols comes by default and which do not etc and other details? Can you provide approx cost range for 100/200/500/1000 virtual users? If one does not want to go with accurate Network simulation by leveraging LoadRunner-Shunra capabilities as mentioned above does LoadRunner offer to simulate Limited bandwidth, Network latency, Packet loss features in mobile network context?

| ‎12-04-2013 09:48 PM

Hi naresh_perf,

 

I've sent you a private message regarding your questions on LoadRunner licensing, etc.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Malcolm is a functional architect, focusing on best practices and methodologies across the software development lifecycle.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation