Data Center Consolidation and move to the cloud: how I tested my applications over WAN

"All the applications in one data center? What about our users in China?" – An angry application owner - HP, 2006

 

I joined HP IT from HP Software late in 2006, after our CIO at the time made a bold move to consolidate HP data centers. This has been much commented in the press, but this move from 85 data centers in 29 countries, down to 6 in the United States was an earthquake for HP IT.
The list of advantages to go through this consolidation cannot be listed in this blog post: it is BIG. But, at the time, I was very concerned that my applications would not have acceptable performance for my customers anymore.

 

Wide Area Network

We know the characteristics of WAN: higher latency – thus longer time for IP packets to reach their destination, possible packet loss leading to retransmissions, jitter and packets arriving in random order. Bandwidth is also usually reduced, although there are techniques to mitigate this. Those impairments are there – no matter how you optimize your network. Don’t get me wrong: you can (and have to!) optimize the network so that is optimal for a data center consolidation. But do not trust the one who will tell you “forget about the network, we will give you the biggest pipes in the world: infinite bandwidth, no latency, no packet loss”. It is just not compatible with the laws of physics. That’s right; speed of light has not been breached so far.

So, in addition to optimize the network, you need to do something about the applications.

HPNetworking.12500.switch.jpg

 

Test performances by emulating WAN performances
To act, I need to be able to understand what’s happening. At the time, I did put a simple test together for each of the applications: test the impact of various network conditions between the application server and the client, for a list of typical actions. This is meant to reproduce end-users accessing a central application instance from various locations in the world.
How to reproduce various network conditions? You have a couple of possibilities there. One is just to use the actual WAN network and run your tests from remote locations, on the live network. It will work, but it will not be very practical or very reproducible. In order to really manage my test environment, I used a WAN emulator. A WAN emulator allowed me to emulate the characteristics of a WAN network, but on a local network.
The setup I used was something like this:


setup.png

 

Then, I selected a set of typical actions in the application and executed the test by changing the latency (and other parameters) on the WAN emulator, to emulate conditions from various parts of the world. This gave me this (actual data – but the names have been changed to protect the innocents):

 

table.png

 

Which I graphed like this:

 

graph.png
 

There are a number of things that you can get out of that kind of graphs. See the data set with triangles? Application response time is very affected by latency under this scenario. This tells me that the application is probably very chatty on the network. You also need to discuss with your users to set acceptable thresholds for each of the actions, and decide what to do next.

 

What’s next?
Wait a minute. What can actually be next? For me, the answer was different for each application.
- Applications developed in house: work with the developers to reduce chattiness and implement other strategies such as caching and asynchronous requests/responses (parallelization is great).
- Applications from 3rd party vendors: a difficult case, as it may be very hard to change the application to implement strategies as the one above. In any case, those changes do take time. There may be quick wins here and there, but be prepared for a multi-months effort with the vendor.

 

What would I do differently today?
This worked. But all the tests were manual and time consuming. For example, I needed to test again after the delivery of a performance patch from our developers.
Since then, we have worked a lot at HP IT to test the performances of our applications using HP LoadRunner and HP Performance Center (this will have to be the topic of another blog post). WAN emulation – and more - is part of the party, thanks to HP and Shunra’s partnership. I can reliably emulate WAN conditions and see how this impacts the performances of my applications. This gives me very nice data points and reports which I can easily throw on the desk (read: “email inbox” in our digital world) of my preferred application owner, so that we can do something about it.

 

And now…
6 years have passed. The datacenter consolidation was completed in 3 years, and is still very much in effect today. This topic though has not gone away. Like many, I am also looking to move some of my applications in the cloud (private and hybrid in my case). Also, more and more of our applications go mobile: network conditions are really hard in the mobile world! Bottom line, our HP Performance Center and HP LoadRunner investment is paying off – day after day.

 

I am really curious on how you test that your applications will not break when you consolidate your data centers, or move to the cloud? Maybe you are using techniques I didn't thought of: let's discuss about it below.

 

footer-discover2012-sw-600x60.gif

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
HP IT Distinguished Technologist. Tooling HP's R&D and IT for product development processes and tools.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation