Cloud automation takes some of the heartburn out of Heartbleed

flood waters.jpg

 

Guest post by Warren Volkmann, Cloud Journey Storyteller

 

A week after news of the Heartbleed bug raced around the world like a cyber-tsunami, I found myself facing my own personal equivalent when the water heater blew out.

“Dad!” my daughter called out, “There’s water everywhere in the garage.”

I ran.

We had moved from a house with a full basement to a house with no basement, and the garage had become the repository of the precious overflow.

 

After cranking on the 40-year-old faucet to shut off pressure to the 10-year-old water heater (1 month out of warranty, of course), I turned to survey the damage.

 

“Heartbleed,” I thought. “This is like my own personal Heartbleed crisis.”

My sudden discovery that everything that I thought was safe and secure had actually been at risk for a long time was pretty much what happened to the entire online world on April 7, 2014. That’s the day the OpenSSL team, which maintains free security software that almost everyone uses, announced that their software had a bug in it.

 

The security advisory, titled “TLS heartbeat read overrun,” seemed innocuous:

A missing bounds check in the handling of the TLS heartbeat extension can be used to reveal up to 64k of memory to a connected client or server.

 

To those who understood security protocols, the unbounded implications were instant and global. The bug was not a virus or malware that could spread maliciously from server to server. It was worse. It was “security vulnerability” that was already installed on almost every server, ready to be exploited by anyone who discovered it. Heartbleed was like a gap in a border fence through which zombies could enter. (Fortunately, nobody has yet detected an invasion, although specters from the NSA have been widely reported.)

 

Anything on the Internet that relied on OpenSSL for encryption – which is practically everything – was unprotected. All those online shopping and banking websites with the padlock icons and “https” URL (“s” for “secure”) had been, in fact, wide open for about two years.

 

It reminded me of the discovery in 2005 that the most popular kind of bicycle lock – a u-bolt that used a circular key – could actually be opened by hammering a soft plastic pen body into the circular key slot.  The soft plastic, under pressure, pushed the pins inside the tumbler until they aligned, forming an instant key. Locks that were thought to be impervious could be popped open with a 25-cent pen. Some thieves stole the bike, then had the gall to relock the lock.

YouTube videos showed how easily it could be done. The company, to its credit, changed the design and sent new locks to anyone who requested them, but that was small consolation to the bewildered cyclist left standing at an empty rack. The security that they had trusted for years turned out to have a hidden flaw, just like Heartbleed.

 

Soggy boxes


Now, most people when they see sheets of water streaming through stacks of possibly precious possessions, don’t think about cyber security. However, I am the imbedded writer for an HP business – Publishing Solutions Operations (PSO) in Corvallis, Oregon. PSO is central to HP’s plans for Internet printing. The eOps team supports several HP cloud-based businesses, including HP Printables (printer apps and cool publications that you can subscribe to through HP’s web-connected printers. Check ‘em out at http://www.HPConnected.com).

 

PSO has deep roots. Many of its managers were HP’s original web service pioneers, veterans of the data center era when technicians spent  days provisioning stacks in racks and configured servers using keyboards that were too high to sit in front of, but too low to stand at without back strain.

 

When HP’s public cloud was ready for business, PSO was a high-profile, internal business that migrated early. I was tapped to document the “cloud journey” and capture the new HP Way of cloud computing.

 

1400 bleeding hearts

 

When Heartbleed appeared, the PSO cloud migration team was faced with more than 1400 servers that potentially needed patching. In the early days of manual configuration management, where each server needed to be vaccinated individually, such a task would have taken weeks – maybe more than a month. This gap left lots of time for the bad guys to figure out how to exploit the bug.

 

Fortunately, when the PSO cloud team migrated their stacks to HP cloud, they employed some of the newest tools available to automate stack creation and maintenance. The team was an early and aggressive adopter of Heat, an OpenStack tool that automates stack provisioning. (Heat is the cloud equivalent of the server technician with his box cutter knife and tool belt who would unpack and rack the servers and switches.)

 

The PSO migration team also employed Puppet Enterprise software (similar to Chef Software) to deeply automate and standardize configuration management. It was Puppet that proved so useful in reacting quickly – nowadays everyone says “agilely” – to Heartbleed. 

 

700+ in 3 hours

 

When the OpenSSL Project issued a new release that would remove the Heartbleed bug, PSO used Puppet to push out the new-and-improved bits to more than 700 servers in about three hours. (Puppet could have done them all in minutes, but the team took time to confirm each update.) Within hours all PSO stacks that had been vulnerable to external attack were protected.

 

Puppet (and other configuration management approaches) are to Heartbleed what an antacid tablet is to heartburn. They don’t cure the entire problem, but they can make the worst of it go away “agilely.”

 

PSO’s response to Heartbleed demonstrated that enterprise cloud computing and automated configuration management fit together tightly to enable scaled-up enterprises to respond and adapt – dare I say – agilely.

With the businesses safe and secure, the team could turn its attention to the hundreds of servers that were not yet automated. They would have to be updated in the good ol’ fashioned way – by hand. Every server on every stack got a cold, hard assessment. What is it? What does it do? Who owns it? Who relies on it?

 

It turned out that the stacks on PSO’s cloud and the stacks of boxes in my garage have a lot in common. Both were long overdue for some judicious housekeeping.

 

There’s a lot of stuff in my garage of questionable value – stuff that just accumulated. Some stuff truly is precious (at least to me) – like my soggy National Geographic magazines. Some stuff was useful or precious in its day – like my daughters princess costume and angel wings – but that day is gone. There are similarities on PSO’s cloud. My gonzo teammate, nicknamed The Terminator, is on a “deleting spree,” pursing abandoned and derelict servers for termination with extreme prejudice.

 

Perhaps I should invite him over to help with my garage this weekend.

Labels: HP cloud
Comments
Cloud Slinger(anon) | ‎05-01-2014 09:24 AM

Heartbleed triage and remediation has been an interesting story for vendors selling cloud computing services. I trust HP is investing in research to show whether or not security maintenance and response is more efficient atop cloud computing infrastructure, versus traditional enterprise datacenters? 

Guest Blogger (HPSW-Guest) | ‎06-18-2014 01:38 PM

HP has more than a thousand employees working on security, and is a leader in a much larger security initiative by the OpenStack community. (HP’s newly announced Helion cloud is based on OpenStack.) Ben de Bont is Chief Security Office for HP Cloud. If you want more specifics about HP’s approach to cloud security, here is Ben’s last security podcast on “Grounded in the Cloud.”

http://h30499.www3.hp.com/t5/Grounded-in-the-Cloud/Podcast-Discussion-with-Ben-de-Bont-of-HP-Cloud/b...

 

Thanks,

Warren

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
This account is for guest bloggers. The blog post will identify the blogger.
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.