HP Cloud Open Sources Cloud Agents - Automation and Orchestration Platform

Two wonderful things happened to me last week: The first was the birth of my son, all 9 lbs 7 oz of him, and the second was getting approval to open source the Prototype of our OpenStack-integrated automation and orchestration platform, Clou....  Just in case my wife is reading this, I’ll state that the arrival of my son was way better, but I’m really excited to share Cloud Agents with you, too, and really looking forward to watching both of them grow up in the big world.

 

After we launched our public beta last year, the Developer Experience team started having conversations about the new DevOps users we were getting.  Many of them were new to cloud, and needed simple solutions to create value for their teams, not complex building blocks to assemble themselves.  These users needed cloud to cloud container synchronizationscheduled server snapshotscontainer or server usage change alertsfull stack software standup, or they wanted a state change like a performance metric to trigger any of those.  They wanted to run logic in the cloud (whether they knew it or not), but they didn’t want to run their own utility VMs, or they didn’t want to pay to maintain a VM just for a task that ran once a day.

 

As we broke down the problem, we realized that the task logic was usually simple.  Most of the complexity came from the need for a scheduler, a notification pipeline, execution environment, and credentials allowing for individual user context.  We realized that if we built a system to handle all that, we could run small, simple scripts on it that would let users do all kinds of interesting automation and integration work in the cloud, without having to maintain infrastructure.

 

Users could script the cloud themselves, and we could offer lots of quick-to-develop agents to tackle specific use cases, with really simple interfaces that new DevOps users could easily grasp.  These little scripts, which sense the state of a users cloud and can act on it on their behalf, started looking a lot like Intelligent Agents, and thus, Cloud Agents was born.

 

We started building prototypes, and eventually got to the code we’re releasing today.  As we started considering bringing Cloud Agents to production, however, it became clear that an Orchestration and Automation solution would soon appear as a core part of OpenStack.  Given that, we made the decision to not take Cloud Agents to production, but rather to Open Source it, and feed some ideas and inspiration to the development of the OpenStack solution.

 

Now that the backstory’s out of the way, here’s a short introduction to Cloud Agents that shows off some of its nifty features:

 
 

Cloud Agents is a PaaS for running small, autonomous programs (we call them Agents) in the cloud.  Right now they’re Python scripts, so they can use the standard OpenStack Python libraries to manage OpenStack resources. The software runs as a user in a tenant, so it receives a Keystone token and tenant ID, and can make API calls and trigger actions on their behalf.  Agent scripts are reusable, so you can use the same cloud to cloud container synchronization script I do.  Agents can be run both in a managed Cloud Agents service in the cloud, or on any UNIX machine, whether it be in your datacenter, a local VM, or in a different cloud.  If you build a Cloud Agent, you aren’t tied to the cloud service, you can run it yourself on your own machines forever.

 

Cloud Agents output their configuration in JSON on demand, and you can ask the service to just give you the configuration options for an agent via the API.  That means you can create a web or mobile interface for agents, doing client-side form validation and other user friendly stuff.  We’ve included a Cloud Agents web UI with the prototype to show how to do that.  The Cloud Agents Python module also knows how to produce an interactive Command Line configuration interface, and can read configurations as JSON.  The output from the agent is delivered as JSON or plain text, for local usage.

 

Agents can be bundled into the agents service runtime, loaded from a user’s Object Storage containers, or retrieved from HTTP or HTTPS URLs.  This means that users can write their own agents, providers can bundle supported agents in, and based on the URL of the agent script, the service can treat them differently, but still have a consistent execution endpoint.

 

The Cloud Agents service supports email notifications to users, and code to allow for sending those via SendGrid is included.

 

The Cloud Agents service is made up of a few components.  It’s a fairly standard scalable task execution platform, and here’s a friendly diagram:

techcon_poster_diagram.png

 

First, there’s a front end API server that clients (web, cli, etc) talk to.  Secure information is encrypted before being stored, and decrypted as needed in a separate crypt service.  A dispatcher pulls tasks to be executed out of the queue, retrieves a key from Keystone, and sends the task out to a distributed cluster of execution servers in the cloud.  The software on those execution servers creates a secure environment for the Agent script, downloads it, and runs it, delivering the output back up the stack to the database, where the client can retrieve it.  We initially built it with a chroot solution for script isolation.  If we were writing it today, it would probably be Linux Containers under Docker.

 

As we built out the Agent use cases, we added a couple of features that I think really make Cloud Agents unique.  First, we exposed the Cloud Agents endpoint and task IDs to the agent script.  This means an agent can modify it’s own configuration and schedule, or even schedule new tasks for itself or other Agents.  A server standup agent can create a recurring backup schedule, usage notifications, an availability zone or capacity resize plan, and more.  Since Agents have access to the whole Internet, servers could be resized based on trending tweets, or metrics from an off-cloud API.

 

The second feature is that we added a private data store for the agent, a small space that they can write to and are supplied with on startup, but isn’t accessible via the API directly.  This means that an agent can generate credentials or other secure information, and keep it between execution runs without worrying about it being exposed to another agent.  This data can be JSON, base64 encoded binary data, or whatever the agent wants.  It provides some really interesting application use cases.

 

Since Cloud Agents is integrated with OpenStack, it’s easy to build non-user-facing applications on top of it without needing a full VM all the time.  One idea that I’m still toying with is a twitter bot that stores discreet data in Object Storage for each user that tweets at it.  It could run every 5 minutes, do any processing it needed to do, respond to the new tweets directed to it, and store a per-user application state in objects in a container.  I was thinking of starting with a Tic-Tac-Toe game, one where you could ask it your win/loss percentage over time, but there are lots of possibilities.

 

It’s really exciting to look at the cloud as a new kind of computer, one where apps run for individual users, not in traditional full-machine VMs, but distributed across it, transient, bursting and storing data where it’s most appropriate.  There are a lot of possibilities there, and I feel we’ve only begun to scratch the surface.

 

The code here is pretty rough.  It’s just a demonstration prototype, but it’s been running across a multi-node cluster in the cloud for a year, and sends me usage notifications and other helpful things every day.  With some developer attention, it could turn into something special, and I’d love to see ideas appear in things like Mistral.

 

Today we’re Open Sourcing Cloud Agents under an Apache 2 license.  You can grab the code on GitHub.  There are instructions for standing up the service on an Ubuntu 12.04 machinesome agent developer documentation, and some sample agents that show the kinds of things the service can do.  As the developer of the code in this release (and someone who is really excited about the idea of Intelligent Agents), I’m also maintaining a personal Cloud Agents fork where I’ll be actively tinkering on it in my free time, and accepting pull requests. 

 

If you have questions or comments about Cloud Agents, please leave them here or on GitHub.  Thanks!  

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Stephen Spector is a HP Cloud Evangelist promoting the OpenStack based clouds at HP for hybrid, public, and private clouds . He was previous...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation