Semaphores in HP Operations Orchestration

In computer science, a semaphore is a variable or abstract data type that provides a simple, but useful, abstraction for controlling access, by multiple processes to a common resource in a parallel programming or a multi-user environment.

HP Operations Orchestration (HP OO) is a next-generation IT Process Automation solution that was designed from the ground up to increase automation adoption—whether in a traditional data center or hybrid cloud environment. It provides the perfect choice for automating your IT tasks, operations and processes.

It also helps you automate the most comprehensive integration capabilities and out-of-box content across multiple IT domains such as:

  • Virtualization
  • Cloud
  • SAP orchestration
  • Dev/Ops
  • Security Ops

HP OO customers are running multiple concurrent workflows and by using semaphores you can ensure, that concurrent flows using a shared resource only access it one at a time.  For example: when updating a storage array or a VCenter instance.

 

In this blog post we will show you how HP OO and semaphores can help you achieve that.

 

 

Overview

The semaphore concept was invented by Dutch computer scientist Edsger Dijkstra in 1965[1], and today is an integral part of most operating systems. The concepts extend to more than just operation systems and apply to most concurrent systems—such as HP OO. Semaphores are useful tools in the prevention of race conditions; however their use is by no means a guarantee that a program is free from problems such as dead locks. To learn more about semaphores, mutexes and dead locks etc. there are many online and resources freely available. I also recommend reading Andrew S. Tanenbaum’s book “Modern Operating Systems” (1992).

In a HP OO a flow would require a semaphore to gain “access” to a shared resource. This would typically include these steps:

  • Wait for the semaphore to allow access to the shared resource
  • Access and use the shared resource
  • Release the semaphore to allow other flows to access the shared resource

The rest of this blog post will be fairly technical and is aimed at HP OO practitioners and those of you who are curious about how automatic workflows are created in HP OO.  We will first explain the concepts of locks in OO, and then I will show two examples using locks as semaphores.

 

HP OO Locks

The semaphores in the following two examples are based on three HP OO out of the box operations and flows “Acquire Lock”, “Release Lock”, and “Wait For Lock” which are found under the library path “/Library/Integrations/Hewlett-Packard/Operations Orchestration/”.

The subflow “Wait For Lock” repeatedly tries to acquire the lock until it becomes available. The flow takes two parameters (in addition to the lock id) to specify a timeout and how many seconds there should be between each attempt to acquire the lock.  This flow is shown below:

1-locks.png

 

HP OO relies on the database to ensure that the acquisition and release of locks are atomic and thread-safe operations.

There are three very important limitations to the Acquire Lock operation:

  1. The operation does not block waiting for a lock to become available.
  2. The locks acquired by this operation are not re-entrant. This means that once a lock is acquired by a run, all subsequent attempts at acquiring the same lock by the same run will fail.
  3. A lock belongs to a flow (by UUID) so two independent flows cannot share the same lock.

These are limitations that we cannot easily overcome, so it is important to remember to acquire the lock before any looping, and release the lock after the loop and in general be efficient when holding a lock. It is important to remember that other operations may be waiting for that lock.

 

Simple Example: Single Shared Resource

This example shows how to use “Wait For Lock” and “Release Lock”. The basic principle is always:

  • Acquire the resource
  • Use the resource
  • Release the resource

In this example, to keep it simple, the shared resource is a file that we want to append some text to. Again to keep it very simple we created an OO RAS command with this command:

  • echo This is flow '${run_id}' writing. >> ${filename}

In your own flows you would replace this command with all the operations that uses the resources (perhaps encapsulated in a subflow).

We use the filename as the lock ID as it is the unique identifier for this resource. The example flow looks as follow:

2-simple-ex.png
 

It is best practice to always release the lock after use. Notice that we have added two optional steps in the beginning of the flow that makes the flow sleep for a random amount of time (1-9 seconds). When we launch five concurrent flows in the test, it will likely result in flows that write to the text file in an out-of-order fashion as shown below:

3-simple-result.png
 
Multiple Resources

The example above showed how multiple running instances of a flow could access a single shared resource. This concept can be extended to handle a list of multiple resources – noting that the resources should be more or less identical. A real life example could be a pool of hosts that are not behind a load balancer.

The HP OO Community team has created a Semaphore Guide which shows in detail the examples in this blog post as well as how to extend the Semaphore example to handle multiple resources. Find this and all the flows on the HP OO Community: https://hpln.hp.com/group/operations-orchestration

 

Advanced Example: Single Global Shared Resource

As mentioned above, one of the limitations with locks is that they are specific to one particular flow even if we use the same lock Id. It is a common problem is to have many different flows use the same shared resource. For example: having many different flows to automatically handle different types of incidents, and each flow will need to perhaps both read and update the incident in the ticketing system, e.g. HP Service Manager.

Let us imagine that the ticket system only accepts 50 concurrent connections, thus we want to put in place a global semaphore to control access to the shared resource (the ticketing system). For this we need two things from OO:

  • A way to keep count of how many flows that are currently accessing the resource
  • Protect this counter from concurrency issues

Handling Concurrency
Starting with the latter point, we know from the example above that OO locks are only applied to the scope of the parent/main flow that obtains the lock (for details please refer to the full guide as mentioned above). In order to ensure that all lock operations belong to the same flow, we will create a subflow that handles all lock operations and launch this as a separate flow using the OOTB operation:

  • /Library/Integrations/Hewlett-Packard/Operations Orchestration/Dynamically Launch Flow

This creates a context that is always the same no matter which flows that tries to use the subflow. As the examples from earlier show, the global semaphore should support Acquire and Release. The flows, which we have made available on the OO Community, contain the following sub-flows used by the global semaphore implementation:

4-flows.png   5-launcher.png
 

The important three subflows are:

  • Acquire Global Resource
  • Release Global Resource
  • Reset Global Resource (explained later in the guide)

Since these are the resources you want to use when working with global semaphores. These resources are all implemented in a similar fashion using the Dynamically Launch Flow as shown above.

 

All of them call the subflow “Global Semaphore Main” but with different inputs. As in the earlier example, we might risk a timeout while working with locks; this is why we check the result returned from the launch operation.

The “Global Semaphore Main” flow ensures all locks belong to the same parent flow. It takes the following five inputs:

  • action
  • s_name
  • max_count
  • timeout (when to give up)
  • seconds (delay between each try)

The ‘action’ input specifies if we want to acquire, release or reset a global resource and the “s_name” is the global unique name of the semaphore e.g. the name of the ticket system or another identifier that is known by all flow authors in your company. The other inputs work as in the examples above. Note that this is a counting semaphore which can behave as binary semaphore if max_count is 1. The “Global Semaphore Main” flow performs the following three semaphore actions (here using the classical semaphore terms):

  • Reset – Used to initialize or reset the counter, which might be needed in case of crash or in case of someone implements a flow that forgets to release the semaphore.
  • Notify – Notifies the semaphore that a flow is done, and that the counter can be decreased. It requires that the counter has been created.
  • Wait – Wait for access to the semaphore, this goes fast if the counter is less than max_count, else the semaphore enters a wait loop based on the specified inputs.  It requires that the counter has been created.
The flow is illustrated below (we see that the flow tests if the counter has been created for the cases of Notify and Wait):
6-global-semaphore.png
 

The Counter
All three semaphore actions acquire a lock before using the counter. We implement the counter using the OOTB operations:

  • /Library/Integrations/Hewlett-Packard/Operations Orchestration/Cross Run Data Persistence/Get Stored Flow Variable
  • /Library/Integrations/Hewlett-Packard/Operations Orchestration/Cross Run Data Persistence/Store Flow Variable

Below we show how the Global Semaphore Wait flow is using these operations:

7-semaphore-wait-subflow.png

 

It includes a wait loop and acquires the locks before working with the counter. The other two actions are implemented in a similar way. The counter is stored in the OO database.

 

Using the Global Semaphore

We created two test flows to show how to use the global semaphores; this is the flow “A flow using the global resource”: 

8-advanced-ex.png
 

This is exactly the same concept as in the simple examples above. BUT it is important to notice that you must always release the global semaphore before exiting the flow, else the counter is not decreased and over time the counter will attain the max value and the global resources protected by the semaphore is no longer usable.

In the test we launch 50 concurrent flows. And if we track the counter, which is stored in the OO database, we observe that it steadily rises to 15 (which is the max_count in our example):

mysql> select flow_uuid, var_key, var_value, tstamp from flow_vars;

+--------------------------------------+---------------------------+-----------+---------------------+

| flow_uuid                            | var_key                   | var_value | tstamp              |

+--------------------------------------+---------------------------+-----------+---------------------+

| 0b9bcf84-4f97-47e6-a7b2-a39ae92fa863 | MyGlobalResource1_counter | 15        | 2013-08-31 01:27:55 |

+--------------------------------------+---------------------------+-----------+---------------------+

And after a while, it decreases towards 0 where it eventually stops when all flows are done.

 

 

The full Semaphore guide is available on the HP OO Community:
https://hpln.hp.com/group/operations-orchestration

 

All the flows can be found on the OO Content Community here:
https://hpln.hp.com/node/46/contentfiles/?dir=16086

 

 

You can also find more information about HP Operations Orchestration here.

 


[1] Dijkstra, Edsger W. Cooperating sequential processes (EWD-123). Center for American History, University of Texas at Austin. (September 1965)

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
HP Software Cloud and Automation solution architect serving largest customers in Europe, now leading the Community team for HP Operations Or...
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.