Tutorial - Using Worker Roles for Simple Background Processing

Author: Craig Kitterman

Date: Tuesday, January 4, 2011, 12:00:00 AM

Tags: Tutorial

Table of Contents

Note:This article pertains to the CodePlex SDK initially released late 2009. The Windows Azure team has since then released a newer version of the Azure SDK for PHP on Github. Please refer to the Windows Azure PHP Developer Center for documentation on this more recent version of the SDK.

Please stay tuned and come back here regularly as we are working on refreshing the tutorials to deliver up to date and useful content for our PHP developers.

Recommended pre-read

clip_image001There are many common scenarios in web development that require processing of information, gathering data, or handling message traffic that can be accomplished asynchronously - meaning in the background while the user is doing other things with the application. A common example of this is sending email or when thousands of users are posting comments on your blog. When you open an account or change your password, often web applications will send you some kind of confirmation email as part of the workflow. This is typically done from the server using SMTP relay. Anytime an application is connecting to an internal service there are times when network issues can cause problems. These problems range from slow bandwidth to high latency to server outages - each having the possibility to cause a connection timeout or just simply take a long, long time.



clip_image002When doing this type of processing, you have two options: to "block" and process the message while the user waits on a response from the server, or to allow the user to simply carry on and queue the work for background processing. (I believe the latter approach is highly preferred by the British Monarchy.)

Windows Azure provides simple tools to make this type of background processing a snap. First is the queue system - a part of Windows Azure storage. Windows Azure queues allow you to store a simple message in a durable FIFO (first-in, first-out) queue. This means that once the message is queued, it is stored securely in the cloud, waiting for someone to do something with it - even if your application goes down for one reason or another. The second piece of the Windows Azure platform that enables this scenario is the "worker role". The worker role can be thought of as a simple program with no user interface (in most cases) that does simple tasks in the background regardless of what your "web role" (the interface for your web application) is doing. It is configured by default to run in an endless loop - to look for work, do the work, take a small rest, and then do it again. This type of run loop works very well for processing message queues. The worker role will check the message queue, de-queue and handle all of the messages, then sleep for N seconds (up to the developer to decide). If there are no messages in the queue, the process just goes back to sleep and waits for its next chance to check again.

Example: Simple PHP Email Sender

In this example, we will expand on the basics of queues and how to use them from PHP; (as seen in Maarten Balliauw's article) then take them into an asynchronous scenario. Let's look at how to do this using Eclipse and the Windows Azure tools for Eclipse.

Prerequisites: Your machine must be configured to use Eclipse Tools for Windows Azure and the PHP SDK for Windows Azure. Please see Getting the Windows Azure Pre-Requisites via the Microsoft Web Platform Installer 3.0" and " Using the Windows Azure Tools for Eclipse with PHP".

Step 1: Create a New PHP Azure Project in Eclipse

Start Eclipse:


Select the "Windows Azure PDT" Perspective by clicking the "Window" button on the toolbar, then select "Open Perspective", then "Other":


Select "Windows Azure PDT" in the "Open Perspective" dialog. Click "OK":


Create the new project by clicking "File" on the toolbar then "New" then "Windows Azure PHP Project":


Give your Azure PHP Project a name like "SimpleBackgroundEmailer". Click "Next":


Since a PHP Windows Azure project requires a web role, one will be created by default. The next step is then to configure the web role appropriately. First, give your web role a name. I like to keep things overly complicated so I called mine "Web". Select also the checkbox that says "Windows Azure Data Storage" - this will ensure that the required libraries from the PHP SDK for Windows Azure are loaded and referenced, allowing simple class based access to the Azure storage objects like the queue that we plan to use.


After configuring your web role, you will be given a project summary dialog which lists all the roles in the project. In this example the web role is going to create the work (email messages to process) so we are not done yet. We need someone or something to actually DO work - so let's create a worker role by clicking on the "Add" button to the right of the Worker roles list (the one on the bottom):


In the "New PHP Worker Role" dialog we must first give our Worker role a name. Again, to make things as confusing as possible let's choose something ridiculous like "Worker". We must next give a name to our startup script. This is the script that is going to get executed when the role is started up in the cloud by the bootstrapping code. All our "work" will be contained inside of it. For this one let's make things simple and call it just "startup.php". Select "Run Startup Script Continuously" to ensure that our script only gets executed once. This means that we will have to handle the endless loop ourselves, which gives us a bit more control. As with the web role, select "Windows Azure Data Storage" and "Use Development Storage" to ensure our project is configured properly for working with Azure queues. Note: For demonstration purposes, we are using development storage, but it is trivial to switch to a "live" cloud based storage account and I will show the code to do this.


The result is a list that contains one web role called "web" and one worker role called "worker". If you had a more complex solution where multiple roles handled different tasks you might want to be a bit more descriptive with the naming, but for the purposes of this example this should be sufficient.


After clicking "Finish" you should see a fresh set of 3 projects in your Eclipse workspace: "SimpleBackgroundEmailer" (which contains the configuration metadata for the all-up cloud service package), "SimpleBackgroundEmailer_Web" which contains the files for our "web" site along with some sample scripts, and "SimpleBackgroundEmailer_Worker" which contains "startup.php", some supporting libraries, and some simple sample scripts. We are ready to write some code!


Step 2: Create a Simple Web Form to Input Email Messages

Right-click on the root node of the "SimpleBackgroundEmailer_Web" project, select "New" then "PHP File":


Give your new PHP page a name - something along the lines of "emailform.php":


The result is an empty php page - our empty canvas:


Since we want this fancy new page to be the page that starts by default when we launch our cloud service package, we need to make a minor configuration change. To do this we need to modify the Web.config file. You may be wondering why there is a web.config file in a PHP web project. If you aren't then just smile and follow these steps blindly . If you are, then I must let you know that this is simply part of the Internet Information Server (IIS) scaffolding that sits behind the PHP runtime. By setting the default file in here, it will be sure to load in the browser when your user hits your site root.

To edit this file in Eclipse simply right-click on the file "Web.config", then select "Open With", then "Text Editor".


Modify the value of the default document name highlighted below to be the same as your email form (in this case, "emailform.php").


We then create a simple HTML form to input email messages, by editing "emailform.php":


   1: <html>
   2:     <head>
   3:         <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
   4:         <title>Send e-mail</title>
   5:     </head>
   6:     <body>
   7:         <form method="POST">
   8:             <h1>Send e-mail</h1>
   9:             <p>This sample will allow you to post a short e-mail message into a Windows Azure queue.
  10:                 This e-mail will be sent out by a background process that processes queued messages and can
  11:                 be started below.</p>  
  12:             <div class="region">
  13:             <h2>Create message</h2>
  14:                 <table border="0" cellspacing="0" cellpadding="2">
  15:                     <tr>
  16:                         <th>Receiver e-mail address:</th>
  17:                         <td><input type="text" name="to" value="" /></td>
  18:                     </tr>
  19:                     <tr>
  20:                         <th>Subject:</th>
  21:                         <td><input type="text" name="subject" value="" /></td>
  22:                     </tr>
  23:                     <tr>
  24:                         <th>Message:</th>
  25:                         <td><textarea name="body" cols="60" rows="5"></textarea></td>
  26:                     </tr>
  27:                     <tr>
  28:                         <th></th>
  29:                         <td>
  30:                             <input type="submit" name="action" value="Enqueue" />
  31:                         </td>
  32:                     </tr>
  33:                 </table>
  34:             </div>
  35:         </form>
  36:     </body>
  37: </html>

Now, that is a pretty form. Unfortunately, it is not going to do anything with that message except throw it away into the Matrix. Since we actually don't want it to get lost into the ether, we need to add some PHP server code, to process the posted message. First we create a storage client, using the PHP SDK for Windows Azure, utilizing the default storage account - which in this case is "devstorage". This Development Storage account is the one created by default on your local machine, when starting the Windows Azure Compute Emulator. This allows us to play around in a cloud-like sandbox without actually setting up a real Azure account. If and when you wish to change this to use a real Azure storage account, you can simply uncomment the second block of code and replace the default $storageClient declaration.

The following lines simply ensure the queue exists - in this case, a queue called "mailqueue"; then we serialize the email message and place it in the queue.


At the arrows: (=>) Plug in real storage account credentials.

   1: <?php
   2: /** Microsoft_WindowsAzure_Storage_Queue */
   3: require_once 'Microsoft/WindowsAzure/Storage/Queue.php';
   5: // Connect to queue
   6: $storageClient = new Microsoft_WindowsAzure_Storage_Queue();
   8: /* Change the above line into the code below when connecting
   9:  * to Windows Azure production storage.
  11: $storageClient = new Microsoft_WindowsAzure_Storage_Queue(
  12:   'queue.core.windows.net',
  13:   '<your azure account name, e.g. mystorageplayround',
  14:   '<your azure account key>'
  15: );
  16: */
  18: // Ensure queue exists
  19: if (!$storageClient->queueExists('mailqueue')) {
  20:     $storageClient->createQueue('mailqueue');
  21: }
  23: // Action to take?
  24: if (isset($_POST['action']) && $_POST['action'] == 'Enqueue') {
  25:     // Pack e-mail fields into an array
  26:     $emailMessage = array(
  27:         'to' => $_POST['to'],
  28:         'subject' => $_POST['subject'],
  29:         'body' => $_POST['body']
  30:     );
  32:     // Put message into queue (serialized array)
  33:     $result = $storageClient->putMessage('mailqueue', serialize($emailMessage));
  34: }
  35: ?>

We can now run this page in the Compute Emulator to ensure that it renders and is working (actually placing messages into the "mailqueue" in the Development Storage account.

Right click on "emailform.php", then select "Run As", then "PHP Windows Azure Page". This will package up the web project into a cloud service, deploy locally to the Windows Azure Compute Emulator, and display the page in the browser window inside Eclipse:


If you see this page, you are making good progress and we are half way there; go ahead and create a simple message or two, and "Enqueue" them:


Now, let's go take a look at our Development Storage account, to see if the messages we created are showing up there. To do this, we can use the Windows Azure Storage Explorer, which shows up as a tab in the bottom of the edit window in Eclipse. Select the "Windows Azure Storage Explorer" tab, then click the "Storage Account" dropdown box. Select "devstoreaccount1" and click "Open".


You should now see three buttons below the "devstoreaccount1" dropdown. Select the one that says "Queue(1)". Inside you should see a queue name listed called "mailqueue". This queue was created by our "emailform.php" page, when you submitted the first email message. It now also contains all the messages that were submitted. If you click on it, you will see, in the upper right-hand side of the storage explorer, a list of queued messages. By clicking on a message, you can see the contents of the message below; in serialized JSON format.


Step 3: Configure the Worker Role to Process the Queue and Send the Email Messages

Expand the "SimpleBackgroundEmailer_Worker" project and double-click the "startup.php" file to edit:


Here we create a simple PHP script that contains the same storage setup code as in the web page, as well as a continuous loop that examines the message queue and processes each message. This is done by using the "getMessages" method and then de-serializing the individual elements of the message (to, subject, body). Once we have the individual message elements, we call the PHP mail() method to send the message, then we delete the message from the queue, so it does not get sent again. Once we have processed all the messages in the queue, we put the worker to "sleep" for 20 seconds, to wait for more messages to queue up, then we do the processing all over again.


   1: <?php
   2: /** Microsoft_WindowsAzure_Storage_Queue */
   3: require_once 'Microsoft/WindowsAzure/Storage/Queue.php';
   5: // Connect to queue
   6: $storageClient = new Microsoft_WindowsAzure_Storage_Queue();
   8: /* Change the above line into the code below when connecting
   9:  * to Windows Azure production storage.
  11: $storageClient = new Microsoft_WindowsAzure_Storage_Queue(
  12:   'queue.core.windows.net',
  13:   '<your azure account name, e.g. mystorageplayround',
  14:   '<your azure account key>'
  15: );
  16: */
  18: // Ensure queue exists
  19: if (!$storageClient->queueExists('mailqueue')) {
  20:     $storageClient->createQueue('mailqueue');
  21: }
  23: //loop forever
  24: while (true)
  25: {
  26:     // Fetch messages (hide them from the queue for 20 seconds)
  27:     $messagesToProcess = $storageClient->getMessages('mailqueue');
  29:     // Process messages
  30:     foreach ($messagesToProcess as $message) {
  31:         // Unserialize body text
  32:         $messageEmail = unserialize($message->MessageText);
  34:         // Send out e-mail
  35:         mail($messageEmail['to'], $messageEmail['subject'], $messageEmail['body']);
  37:         // Mark message as processed
  38:         $storageClient->deleteMessage('mailqueue', $message);
  39:     }
  41:     //sleep for 20 seconds
  42:     echo('sleep for 20');
  43:     sleep(20);
  44: }
  45: ?>

Note: For simplicity, we are using the default PHP mail() method to send the messages, which will work only with SMTP relay servers that do not require authentication. For more complex mail scenarios (SMTP with authentication, etc.) you can use a more sophisticated library, like PHPMailer (http://sourceforge.net/projects/phpmailer) or the PEAR Mail package (http://sourceforge.net/projects/phpmailer). If you have a local SMTP that does not require authentication, you must set your SMTP host name, port and "from" user in the php.ini file. To do this expand the "php" folder under the "SimpleBackgroundEmailer_Worker" project, then double-click "php.ini"


In the [mail function] section, modify the highlighted entries to match your local server settings:


Step 4: Run the Project

To package and deploy the entire service to the Compute Emulator, right-click on the "SimpleBackgroundEmailer" project, then select "Windows Azure", then "Run Windows Azure PHP Project in Development Fabric".

Note: If you have not already started the Compute Emulator and Storage Emulator, you will get an error when Eclipse tries to deploy the project. If this happens, you can start the Compute Emulator by simply clicking the windows start button and typing "Compute Emulator" and clicking "Enter" on your keyboard. You can then do the same with "Storage Emulator". For more information on getting the Windows Azure Compute Emulator up and running on your machine, check out Ben Lobaugh's article here: ( /articles/getting-the-windows-azure-pre-requisites-via-the-microsoft-web-platform-installer-30).



I hope this is useful and demonstrates how simple it is to get up and running with a "worker" role.
I look forward to your feedback!

blog comments powered by Disqus

Related Content