Emulators vs. Real Devices for Mobile Application Testing - The Answer Might Surprise You

A guest blog by Gidi Pridor, Perfecto Mobile

 

To support their business-critical mobile applications, enterprises require sophisticated mobile testing solutions that allow them to deploy and support their mobile application development in a timely and cost-effective manner, while reducing risk exposure.

 

Mobile emulators are powerful tools for developing mobile applications and are widely used for both manual and automated mobile application testing. On the other hand, since mobile applications are used on real handsets and not emulators, testing on real devices during the QA process is required to ensure the highest level of application quality.

 

The pros and cons of each of these approaches must be carefully considered by enterprises when formulating their mobile testing strategy. Here are some of the main advantages and disadvantages of each approach:

 

Mobile Emulators: Pros

  • Price - Mobile emulators are free and provided as part of the SDK with each new OS release.
  • Simple - Just download the software, install on your PC and you're ready to go. Multiple emulators can be run in a simple and straightforward manner.
  • Fast – Since emulators are simple client software that runs locally on your PC, they have less latency than real devices connected to the local network or in the cloud.

Mobile Emulators: Cons

  • Increased Risk - With emulators, by definition, you are not testing on the same platform and network used by your users. This means that even if all goes well, you cannot be 100% sure it will actually work on a real device ("false-positives").
  • Hardware and Software Differences – Emulators are typically a "plain vanilla" version of the OS and often do not reflect the specific hardware and software features of each supported device. In addition, as time passes following an OS version release, the emulators are not updated to reflect new devices available in the market.
  • Different network environment - In terms of network configuration, mobile emulators run on the PC (through any personal firewall), connect to the LAN and access the Internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the Internet. These differences could affect application behavior.
  • Differences in Computing Resources – Depending on the processing power of the PC running the emulator and the type of handset or smartphone being used for testing, performance on the emulator may be unrealistically good or bad.
  • No way to test network interoperability – It is important to test the impact of network-related events (e.g., incoming call, text message, etc.) and different network technologies (e.g., HSPDA, WCDMA, UMTS, LTE) on mobile app behavior. Since emulators are not connected to the mobile network, they do not support interoperability testing.

Real Devices: Pros

  • Reliability - Testing on real handsets always gives you accurate results (no false positives or false negatives).
  • Interoperability testing - Real device testing is typically performed in a live network.
  • True user experience - Testing on real devices is the only way to truly understand the user experience, taking into account the CPU, memory, screen size, etc. for a given device.
  • Performance testing - Easier to expose performance defects with real handsets, as well as defects which are the result of the handset itself or its environment.

Real Devices: Cons

  • Logistics and costs - Best practices call for testing on up to 30-40 devices in your market and replacing ~30% of those each quarter to stay up to date. The efforts involved in procuring and managing these devices are significant.
  • Cumbersome for development - In the initial development stages, real handsets are harder to connect to IDE than emulators, which can slow down the debugging process.
  • Security issues - If you are using devices connected locally to your workstation, you will need to make sure USB ports are open. In addition, mobile devices can easily be stolen, allowing unauthorized access to your internal network resources.

Taking into account the pros and cons of mobile emulators and real devices, it would appear that the optimal mobile testing solution for enterprises is not "either-or" but rather "a combination of both."

 

Emulators are very suitable for the initial stages of application development. However, to avoid the costly scenario of releasing a business-critical application with defects, enterprises should perform the major part of their mobile testing on real devices before commercial availability. Each organization will need to determine at what stage to introduce real devices, how many devices are sufficient to cover market needs, and how best to manage those devices. Best practices would indicate that actual development should use emulators (and a few reference real handsets), while sanity and regression testing should be done on real handsets.

 ---------

 

ps - remember to register for our Mobile Month Q&A webinar on the 31st of Jan


If you’re interested in HP LoadRunner or Performance Center you can follow us
- on the HP Performance Validation Blog  or even follow us @hploadrunner on Twitter

And if you’re looking for functional testing – you need to check out The Future of Testing Blog

Of course- go to www.hp.com/go/mobiletesting for all things related to HP mobile testing

Comments
Outsourcing Governance Audits | ‎03-20-2012 02:41 AM

Thanks so much for this post. There is very good and helpful information in this post. Keep up the good work.

Also go to the link below to test new mobile application for iPhone and iPad, named ‘GlobalSourcer’.

<a href= http://itunes.apple.com/us/app/globalsourcer/id488714097?mt=8 title=" Outsourcing Governance Audits "> Outsourcing Governance Audits </a>

Saravanan B | ‎04-12-2013 01:24 AM

Hi,

Thanks for the post

 

http://emulator.mobilewebsitesubmit.com

 

Mobile Emulator is an useful online tool to check the screen resolution on different mobile and tablet devices.

Do your mobile screen resolution check for developed mobile website straight from your desktop.

 

Mobile devices supported:

BlackBerry: 320x240
iphone4s: 320x480
iphone5: 320x568
Galaxy S3: 360x640
Lumia 900: 480x800

 

Tablet devices supported:

Kindle Fire 7: 533x853
Google Nexus: 603x966
Apple iPad: 768x1024

 

-sara

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.