For mobile application testing, do you head to the wild or to the lab?

Please suggest a good testing tool for Mobile.”

LinkedIn is filled with such discussions. The assumption behind the question is that the traditional testing rules still apply. Of course, they do, and of course, the question is legitimate, but only to a certain extent.

 

As testers and QA professionals, we need to ensure that the content we deliver to our customers meets their needs regardless of the where, when, how and what they access.

 

As testing organizations, we should consider two types of practices:

in the wild testing and in the lab testing.

 

In the wild – or crowdsource - testing intermediaries offer to capture the “wisdom of crowds’’ (e.g. Wikipedia) by recruiting, training, certifying and managing demographic segmentation and provide both defect reports and recommendations as part of their value add.

 

Whereas “in-the-lab” testing is what you know as testing activities that take place through your organization (or outsourced) as part of your application lifecycle.

 

Is in-the-lab testing the only way?

 

The main challenge that in-the-lab practices fail to address is how to fit user experience into the overall testing effort. How to identify new defects beyond the scripted exploration? How to address research questions (e.g. A/B testing)?

Dev/Test staff is highly qualified to conduct complex and technical testing procedures, however routine processes can sometimes lead to overlooking potential rote defects.

 

Let’s diagnose this further, starting with the mobile testing disciplines:

 

Mobile testing discipline

In the wild

In the lab

Functional

  • Validation of functionality
    • Smoke / Regressions Testing
    • Offline access testing
    • Negative Testing
  • Peripheral Testing (think Nike Fuel Band)

Helps validate the perception of functionalities through a demographically segmented crowd of testers (pick the right testers) to mirror target users.

Can reproduce a very large number of scenarios.

Caution with using your own staff as repeated testing lead to fatigue, monotony and indifference

Caution with opening non-production application to outside and non-badge employees

Vital with continuous testing.

Ideally suited for regression testing.

Caution with costly onsite device management: think about a combination of on premise (core devices) and cloud deployment (scalability).

cannot reproduce every possible combination.

Check how HP Sprinter for Mobile accelerates manual testing

Non Functional

  • Performance testing
  • Varying network conditions
  • Different network Types
  • Services

Ability to gauge the level of performance that an app will have in 2G, 3G, and even 4G environments. Rather difficult to capture and report on.

Check how HP Network Virtualization for Mobile (Shunra) factors in network data in the context of mo...

Interoperability (IOP)

  • Voice / SMS interrupts
  • Notifications
  • Battery /Cable Removal

Tedious to reproduce (e.g. ask a friend to phone you, have a second phone in hand)

Great fit.

Automation using HP UFT Mobile addresses this use case.

Memory Leak

  • Memory Usage
  • Memory Leaks
  • Garbage Collection

Not fit for purpose

Vital

Installation Testing

  • New App Install
  • Uninstall and Reinstall
  • Upgrade testing

Complementary

Vital

Automation using HP UFT Mobile fully addresses this use case

Context-driven Testing

  • Translation
  • Images and Text
  • Currencies, time zones etc.
  • Location-based services

Vital to validate appropriate terminology and accurate translation

Access to multiple environments at once (e.g. home, office, on the move)

Access to “dirty environments”

Can cover some aspects (e.g. currencies exchange rate, time zone)

Most in-the-lab solutions do not support GPS injection for context switching use cases

Performance Testing

  • CPU Usage testing
  • Network Usage
  • Activity Render time

Not fit for purpose, or difficult to coordinate

Vital

HP LoadRunner with HP Network Virtualization fully addresses this use case

Security Testing

  • OWASP Vulnerabilities
  • Dynamic Testing
  • Static Code Analysis
  • Data Encryption

Not fit for purpose

Vital

Usability and Compatibility Testing

  • User Experience
  • Competitive Analysis
  • Expert Review

Vital for validating the layout and design (A/B testing) according to best practices and industry standards.

Access to multiple environments at once (e.g. home, office, vacation)

Benefit from an outside UX expert

Limited to assess subjective thinking.

 

 

Overall, you have to have an in-the-lab testing practice. This is even more relevant as you shift testing left. Nothing will replace your expertise and your ability to use, reuse and scale your valuable business and testing knowledge.

 

And, as Meg Whitman once stated: “Sometimes, it’s a little bit like being a politician. We have work to do in understanding our users’...”.

In-the-lab practices will sometimes not suffice to address the numerous challenges the world poses.

 

However, do not fall for the easy to understand but flawed argument that in-the-wild testing is a cheap alternative to in-the-lab testing.

It is actually not an alternative at all but a complementary practice.

 

Don’t risk your “Brand” and create additional risks.

 

Think of how to use each practice type to best effect.

And watch your mobile application quality and customer satisfaction soar.

 

/Antoine

 

Want to learn more?

I encourage you to visit our Mobile Apps Solutions homepage to learn how HP solutions can help.

 

Labels: mobile testing
Comments
Maggie Milton | ‎05-27-2014 06:26 AM

The main challenge that in-the-lab practices fail to address is how to fit user experience into the overall testing effort.

I fully agree with this statement, thorough testing in the lab can results in an excellent product, from a functional point of view, but you're only halfway through. You need not only a functional product, you need a user friendly one, and for this you have to go "in the wild".

Alis Nikol | ‎07-22-2014 02:10 AM

 

Sometimes the bugs initially found in mobile app testing can appear small and insignificant, whereupon deeper investigation uncovers bigger problems.” I have been in situations where non-testers look at these “insignificant” probing but were surprised when I later on pointed out big issues after my investigation was through. Now you may choose “wild” or “lab”.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.