Can your e-Commerce website handle one million concurrent users, or even 100 K users? Find out!

Let me take you on a journey through the intense world of large scale performance testing. It involves the power of HP SaaS, performance testing and the approaching holiday shopping season.

 

I would like for you to consider how online retailers make sure their website can handle anticipated traffic during holiday shopping or when they release a new limited edition product. The online retailer is anticipating selling over 100,000 units on the first day. If their web portal goes down from being overloaded, it could result in millions in lost sales and a PR nightmare. To prevent this situation, the company has reached out to HP to simulate real user load and find out any potential issues before actual users hit the website.

 

Here is a look at the assignment:

 

Time: 1 AM EDT

Location:  Virtual (Communications handled by conference call and Webex)

Event: Performance Assurance of online site before peak holiday season

Time on hand: 4 Hours

Total number of people on phone: About 20 Project Manager/ Directors, Sys Admins, DBAs, App Admins, Developers, Hosting team and Performance Engineers.

 

Goal: To reach an expected peak load of users on an application and sustain the load for about 2 hours.

This situation seems like it is occurring in NASA’s control center when they are about to launch a new rocket into space. While there are no rockets being launched in the above situation, the pressure is similar. To perform this exercise, multiple teams across different companies and locations have worked diligently to reach this point. There is a small window of opportunity to test real load on a production instance of application. In 4 hours, the goal is to put expected real world user load on the system, and find any potential issues before any real users find them.

 

Why is this happening?

The three big trends in IT i.e. Cloud, Mobile and Big data are shifting the way applications are written and architected. Applications are becoming more composite and the application components run across different datacenters, including cloud and span multiple systems like ERP/CRM. As a result, we are seeing an increasing number of companies who want to do performance testing on production environment at expected load levels. So the three big trends in IT are resulting in:

 

1.       Increased number of users accessing e-commerce sites due to the shift in mobile

2.       Change in architecture of applications due to cloud adoption

3.       It’s very hard to replicate the production environment in the lab and simulate real world load

 

How can you make it happen?

If you are planning a similar scenario of running large scale performance testing, here are my recommendations:

  1. First you would need performance testing platform (infrastructure and licenses) to support increased load levels.
  2. Shift the load to locations outside the company firewall because real users will be coming from outside, not within the lab

locations.png

In my previous blog I covered how HP SaaS can help you get additional infrastructure for short duration and also gain the ability to generate load from an external location

3.       A special skill set and knowledge base is required to run load tests beyond 50K

  

Best Practices

Traditional performance testing is done in scaled down version of the application. A staging or test environment is created usually with ½ or 1/5th of the capacity of production. While that approach works for smaller load levels, it’s hard to extrapolate results for real world scenario.  To prepare for large scale testing, you should consider following best practices:

 

  1.        Based on the real usage pattern, select the window of test where it will have least amount on impact on end users e.g. 2 AM EDT to 6 AM EDT.  If you have the ability to redirect the traffic to a backup site during testing, do that to minimize the revenue loss. Disable any DDOS alerting/monitoring during the load test.
  2.        Create a VugGen script with best practices as recommended by HP.  This script should include proper think times and correct use of parameters. Try to keep unique transactions to a minimum because it will impact analysis time once test is finished.
  3.        Do a warm-up of the system and then run it full throttle. Typically we ramp up few users and then pause the ramping up to ensure those few users have gone through entire cycle of script. This process helps makes sure that every area of application under test has been accessed after a system refresh. For example, compilation of JSPs so that the initial response time does not skew the results.
  4.        Data cartridges—In most applications, once you have used it, there is no way to recover it. So in case something goes wrong, you need to have the ability to quickly re-load new data into scripts or have your DBA refresh / recycle data quickly so that you can re-use your original data.
  5.         It’s always recommended to initialize all users in advance before executing a test if you have aggressive ramp up times. Let’s say, you want to have a ramp rate of 60 users per sec or 3600 users/min entering your website, you can clog the load generator with initialization. Each initialization would take CPU and Memory resources and may launch a new process on the load generator. Having all processes pre-launched and waiting for payload minimizes the impact of ramp-up on existing users that are running on Load Generators.
  6.        Monitor, Monitor, Monitor—Closely monitor your controller and Load Generators to ensure the CPU or memory is not spiking and remaining at high level. This reaction can skew your response times. As a best practice, the CPU and Memory utilization on load Generators never exceed 80% If you see CPU going higher than that, try adding more Load Generators.
  7.       Ensure all of your Load Generators and controller have enough disk space and memory. This is helpful in case you lose connectivity in the middle of run, you have to ability to quickly add additional LG and reset the users with new data
  8.        Always communicate with the team on numbers reported by Performance Center versus what the admins see on their side during test. You will need to report on the number of users in the system, transactions per second, CPU/Memory and other metrics on backend. Validate that the load is properly distributed among all web and app servers. The concurrent uses reported in the Performance Center correlates with backend active sessions and users logged on.
  9.        If for some reason, the test has to be stopped in middle of execution, do not collate or analyze results if you are planning to run another test immediately. Opt for late collate and analyze.

Conclusion

While this is the list for Performance Center best practice, each team from Sys Admin, DBAs and App admins should have their own list of things that they can verify in advance and be ready when D-day comes. In my previous roles at HP SaaS as a Performance Engineer and Customer Success Manager, I have successfully survived these “high-intensity” load testing sessions. It’s an ultimate adrenaline rush for techies to see the application buckling under the load that we are generating. It’s one of the very rare moments when you will see competitors in a single meeting with a single goal in mind—to make a test as realistic as we can. The ultimate goal is to uncover any real issues that can potentially impact revenue during the busy holiday season for the online retailor.  

 

The HP SaaS team has been running large scale tests for over a decade.  We have served customers from every sector who anticipate 100K+ user loads on their applications and want to test on production environment. Performance Center is leader in performance testing as it supports wide range of protocols e.g. Java/JMS, Oralce DB, Web services/SOA, SAP etc.   So whether it’s an ecommerce site, or a headless webservice or JMS, Performance Center on HP SaaS have it covered.

 

Not only do HP SaaS provide infrastructure and licenses, but also the guidance and knowledge of handling large scale testing. On ongoing basis, we run multiple six-figure user load tests for our large retail customers and have successfully tested up to 1 million concurrent users for internal HP testing.

To summarize, just like NASA’s rocket launch, each team has to put lot of efforts to mitigate risks that may come when you actually execute large loadtest.

 

Find out addtional ways HP SaaS can help you, visit our homepage

 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.