Tutorial - Using Queue Service

Author: Maarten Balliauw

Date: Tuesday, October 26, 2010, 12:00:00 AM

Tags: Tutorial, Storage

Table of Contents

Note:This article pertains to the CodePlex SDK initially released late 2009. The Windows Azure team has since then released a newer version of the Azure SDK for PHP on Github. Please refer to the Windows Azure PHP Developer Center for documentation on this more recent version of the SDK.

Please stay tuned and come back here regularly as we are working on refreshing the tutorials to deliver up to date and useful content for our PHP developers.

Recommended Reading

Synopsis

This tutorial will show the fundamentals of Windows Azure Queue storage and provide a walk through that creates a working guest book application that can be deployed directly to Windows Azure.

Sample Files

This tutorial uses the GuestBookUsingQueues sample application from the Windows Azure Sample Kit 4 PHP

Place a copy of the GuestBookUsingQueues files in C:\temp for the remainder of this tutorial.

The file from GuestBookUsingQueues which will be focused on in this tutorial is index.php.

Windows Azure queue service fundamentals

Windows Azure queue provides a reliable message delivery mechanism. It provides a simple asynchronous work dispatch mechanism, which can be used to connect different components of an application. The Windows Azure Queues are highly available, durable and performance efficient. Its programming semantics ensure that a message can be processed at least once. Any application that has the correct credentials can access the queue at anytime from anywhere in the Internet via the web.

A queue can contain an unlimited number of messages, each of which can be up to 8 KB in size. Messages are generally added to the end of the queue and retrieved from the front of the queue, although first in/first out (FIFO) behavior is not guaranteed. If you need to store messages larger than 8 KB, you can store message data as a blob or in a table and then store a reference to the data as a message in a queue.

Addressing schema

A Windows Azure queue is part of a storage account and is structured as follows: an account can contain zero or more queues, which can contain zero or more messages.

Queues are available on their specific HTTP(S) endpoint:

This endpoint is specific to the production Windows Azure environment. When working locally one can install the Windows Azure SDK and have access to a local, simulated storage environment. The endpoint for this environment is:

Access to the storage endpoints are granted based on an account key that you choose and a generated account key. For the development storage environment the following account is the default one:

Account name: devstoreaccount1
Account key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

There is no need to remember the above: the Windows Azure SDK for PHP uses the above account name and account key by default when connecting to development storage.

image

Figure 1: Windows Azure queue structure

Architecture with queues

Consider an online photo hosting service application as an example. Users can upload pictures to this service after which the pictures are resized into various sizes (thumbnail, small, medium, large, original). One would build this application architecture with one or more web servers in front, serving the actual web application and accepting uploaded pictures. A number of back-end servers would also be deployed to resize uploaded images into various sizes. Communication between both sides can be achieved using queues as demonstrated in the following diagram:

image

Figure 2: Queue architecture

Using this architecture there are a number of advantages in using the Windows Azure queue service:

  1. Scalability - both front-end servers and back-end servers can easily be added, pushing messages into the queue and processing messages from the queue.
  2. Reliability - every message in the queue is guaranteed to process. If a few backend servers crashed, instead of losing all the work items, the queues can buffer all the work items sent while the backend servers are down. Messages being processed by the back end servers when they crashed are also not lost as they reappear in the queue after a specific amount of time if they have not been marked as processed by the back-end server.
  3. Decoupling - front-end and back-end are decoupled and can scale independently from each other. Add more front-end servers to accept more image uploads, add more back-end servers if the image processing takes too long.
  4. Buffering - As traffic increases and decreases, queues provide a buffer between individual components. During heavy traffic the queue may grow longer and buffer messages but will eventually be processed when traffic cools.

Queues in this sample

To demonstrate working with queues, a sample scenario will be used. This sample will allow you to post a short guestbook entry into a Windows Azure queue. This message will later be processed by a background queue message processor.

Connecting to Windows Azure queue storage

The Windows Azure SDK for PHP supports connecting to queue storage both in the Windows Azure cloud and locally for development purposes.

Local storage

The Windows Azure development environment includes a local version of the Windows Azure storage server. When the Microsoft_WindowsAzure_Storage_Queue object is not provided connection details in the constructor it will automatically assume the developer wishes to connect to the local storage account. The following code is used to connect to the local storage table.

$queue = new Microsoft_WindowsAzure_Storage_Queue();

Windows Azure cloud storage

When using the Windows Azure cloud storage the Microsoft_WindowsAzure_Storage_Queue object accepts three parameters in order as follows:

  1. Address to the storage queue server
  2. Storage account (endpoint)
  3. Storage account key

The following code will allow you to connect to a Windows Azure table storage account

$queue = new Microsoft_WindowsAzure_Storage_Table( 
    'queue.core.windows.net', 
    <storage account (endpoint)>, 
    <storage account key> 
);

Ensuring a queue exists before use

If an operation is performed on a queue that does not exist a fatal exception will be thrown and your application will perform unexpectedly, therefore it is always good practice to ensure that the queue in question exists. This can be done quite easily with the following single line of code:

$queue->createQueueIfNotExists('<name of queue>');

In the case of our sample the name of the queue is guestbook.

Insert a new queue item

Queues can hold any sort of message as long as it is first converted to a string. This allows your application to serialize and insert object into the queue. In this sample an array containing two values will be stored.

To insert a new queue item the following code is used

$queue->putMessage(<queue name>, <object to insert>);


In this example two values from the $_POST array will be serialized and stored. The code will look like the following:

$obj = array('GuestName' => $_POST['NameTextBox'], 'Message' => $_POST['MessageTextBox']);
$obj = serialize($obj);
$queue->putMessage(QUEUE_GUESTBOOK, $obj);

Delete a queue item

To delete an item from the queue you first need to retrieve it and provide it to the delete method for reference.

$msg = $queue->getMessages(<queue name>); 
$queue->deleteMessage(<queue name>, $msg);

Note: When deleting items from a queue getMessages must be used, peekMessages will not work in this case.

Deleting all queue items

If you wish to empty an entire queue there is a convenient clearMessages method which will delete all messages from a queue without the need to retrieve each message.

$queue->clearMessages(<queue name>);

Retrieving queue items

Items retrieved from the queue are returned as objects. The original content can be found in message->MessageText. The following properties are available to the object.

MessageId

A unique message identifier.

InsertionTime

The time the message was inserted into the queue.

ExpirationTime

The time the message will automatically be removed from the queue.

PopReceipt

Used for verification when deleting a message from the queue.

TimeNextVisible

Next time the message is visible in the queue.

DequeueCount

Number of times the message has been dequeued. This value is incremented each time the message is subsequently dequeued.

MessageText

The message text, up to 8kb.


There are two methods to retrieve queue items, each with their own pros and cons.

getMessages

Get messages is what you will normally use when processing a message. There are certain methods, such as deleteMessage that rely on the object pass back from getMessages.

When using getMessages all the items that are retrieved are set to hidden in the queue for 30 seconds. They are not removed until deleteMessage is called. This allows an application time to process the queue item. If after 30 seconds there is no other activity the item will reappear inside the queue and can be picked up by another processing application. This is a highly beneficial feature of the queues that allows for the possibility of a processing application to fail without losing any of the queue items it was working on. Later another processing application can come along and perform the work that failed in the previous attempt.

getMessages can be used as follows:

$queue->getMessages(<queue name>, <number of messages to retrieve>);


Note: A maximum of 32 items may be retrieved with each call.

peekMessages

If your application only needs to look at what messages are in a queue peekMessages should be used. peekMessages allows the application to pull up a list of items in the queue and inspect them without the items being hidden from other agents that may be accessing the queue. The sample provided with this article uses peekMessages to show a listing of items in the queue before processing.

peekMessages can be useds as follows:

$queue->peekMessages(<queue name>, <number of messages to retrieve>);

Run the sample

If you have been following along in the sample code you will have noticed that most of the methods listed in the tutorial have been implemented in order to present you with a complete sample that is ready to be run in your local Windows Azure development environment or in the Windows Azure cloud with very little change.

At this point you will want to run the sample in the local development environment. This can be easily done with the following:

  • Open a command line prompt
  • Type the command 'package create -in="C:\temp\GuestBookUsingQueues" -out="C:\temp\GuestBookUsingQueues\build" -dev=true'

After running this command your default web browser should open and the GuestBookUsingQueues application will be presented to you, running on your local machine.

 
blog comments powered by Disqus