Recommended pre-read
- Understanding the Windows Azure Platform
- Get Started with the Windows Azure SDK for PHP
- Tutorial - Using Queue Service
- Tutorial - Using Blob Storage
Code Sample
Scenario description
Many custom-developed applications need to send email, whether it is a part of the registration process, a way of notifying users when important events occur or anything else. In a classic PHP application, this is achieved by calling the mail() function or a class library relying on that function. This function requires access to an SMTP server or a Sendmail clone located on the system. (Sendmail is a general purpose internetwork email routing facility that supports many kinds of mail-transfer and -delivery methods, including the Simple Mail Transfer Protocol (SMTP) used for email transport over the Internet. - http://en.wikipedia.org/wiki/Sendmail )
Out of the box, Windows Azure does not support either of those. For good reasons, though: since Windows Azure distributes IP addresses to services that may already have been used by someone else before, chances are that the IP address sending out e-mail may be blacklisted and mail will end up in the recipient's spam folder.
There are some options for sending out e-mail from Windows Azure:
- Using an on-premise mail server for sending out e-mail.
- Using a third-party SMTP relay service. Check http://www.bing.com/search?q=smtp+relay+service for a list of service providers.
When using option 1, the on-premise mail server should either be publicly accessible for sending SMTP. In this scenario, the on-premise mail server is not exposed to the Internet and a different mechanism has to be developed to push e-mails from Windows Azure to the on-premise mail server.
As demonstrated in the article entitled, Tutorial - Using Queue Service, a Windows Azure queue could be leveraged to asynchronously communicate between 2 applications. E-mails might get large and a queue message may not provide sufficient storage for the e-mail message: a queue message can only contain a payload <= 8 KB. E-mails with attachments, rich HTML mails and other large e-mail messages will probably not fit within this storage limit. By combining the queue service with another storage service however, it should be possible to communicate larger payloads. The following options are available:
- Blob storage
- Blob can be up to 200 GB in size (block blobs)
- Blob can be up to 1 TB in size (page blobs)
- Table storage
- One entity can be up to 1 MB in size
- Data distributed over a maximum of 252 properties, each up to 64 KB
- Queue service
- Message can be up to 8 KB
The ideal architecture would be that a queue contains messages with a unique identifier as their payload and that this unique identifier references a blob or table entity. Because table storage space per entity is still limited, this scenario will be combining queue service with blob storage.
Figure 1 Scenario architecture
For this scenario, a simple user registration page will be developed. This login page sends out a confirmation e-mail through a Windows Azure queue message and a Windows Azure blob. On the on-premise mail server, a script polls the queue for new messages and sends the message out.
Figure 2 Scenario screenshot
In the script running in the background, you will see the following log entries:
2010/11/18 09:40:09 No messages found. Backing off for 1s.
2010/11/18 09:40:10 No messages found. Backing off for 2s.
2010/11/18 09:40:13 No messages found. Backing off for 3s.
2010/11/18 09:40:16 No messages found. Backing off for 4s.
2010/11/18 09:40:20 No messages found. Backing off for 5s.
2010/11/18 09:40:25 Found 1 messages to send.
2010/11/18 09:40:25 Backoff value has been reset.
The polling script will use a "backoff technique". If the script would poll the queue a few times every second and no messages are present, a lot of unnecessary queue transactions are performed (and billed!). To overcome this problem, the script polls the queue. If a message is present, the script processes it. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds. Next loop 3 seconds, four seconds, … up to a maximum backoff limit. If messages are present at a given time, this value is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of "exponential backoff", but we will not be using that for this scenario. (Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate. - http://en.wikipedia.org/wiki/Exponential_backoff )
Communicating between applications and application components through queue service
The scenario consists of two separate processed: a registration page and a CLI script running on the on-premise server.
Running the application
Registering for an account
When navigating to the web application, the main screen showing an account registration form is presented. It allows you to register for a user account, an action which will generate a welcome message via e-mail. This message will be added to blob storage and a reference will be added to a queue. The message will then be sent out by a back-end process on a different server.
Figure 3 Main screen (registration)
Sending out e-mail
The back-end process will run as a CLI application. It polls the queue using a backoff mechanism and sends out the e-mail stored in the queue service and blob storage whenever a message is present in the queue.
Figure 4 Back-end script polling the queue
Installing the sample scenario
The sample scenario demo application consists of the following files:
- scenario_interapplication.php
- scenario_interapplication_mail.php
- scenario_interapplication_process.php
These can be deployed and used on any web server (Apache, IIS, …) and platform (Linux, Windows, …).
In order to run the samples out-of-the-box, ensure development storage is started on your Windows machine by clicking Start, All programs, Windows azure SDK, Development Storage. Refer the Tutorial - Getting started for installing Windows Azure SDK for PHP and related components.
Running the sample in the Development Storage or in production with a Windows Azure Storage account
Throughout the scenario, development storage will be used for building examples unless specified differently. If you decided to work with a production Windows Azure storage account, use the production parameters for all constructors of the Microsoft_WindowsAzure_Storage_* objects. If the tutorial connects to the development blob storage service like this:
1: $storageClient = new Microsoft_WindowsAzure_Storage_Blob();
2:// change that code into:
3: $storageClient = new Microsoft_WindowsAzure_Storage_Blob(
4:'blob.core.windows.net',
5:'<your azure account name, e.g. mystorageplayround',
6:'<your azure account key>');
Coding the application
Registering for an account
When navigating to the web application, the main screen showing an account registration form is presented. It allows you to register for a user account, an action which will generate a welcome message via e-mail. This message will be added to blob storage and a reference will be added to a queue. The message will then be sent out by a back-end process on a different server.
The following is an extract of the HTML code displaying the main application screen:
1:<divclass="main">
2:<h2>Description</h2>
3:<p>...</p>
4:5:<?phpif ($registered) { ?>
6:<h2>Thank you for your registration!</h2>
7:<p>A confirmation mail has been sent to your e-mail address.</p>
8:<?php } ?>
9:10:<formmethod="POST">
11:<h2>Register for an account</h2>
12:<fieldset>
13:<legend>Account details</legend>
14:<tableborder="0"cellspacing="0"cellpadding="2">
15:<tr>
16:<th>Your name:</th>
17:<td><inputtype="text"name="name"value=""/></td>
18:</tr>
19:<tr>
20:<th>Your e-mail address:</th>
21:<td><inputtype="text"name="email"value=""/></td>
22:</tr>
23:<tr>
24:<th></th>
25:<td>
26:<inputtype="submit"name="action"value="Register"/>
27:</td>
28:</tr>
29:</table>
30:</fieldset>
31:</form>
32:</div>
This is basically a no-code page. However, when a user clicks the Register button, an e-mail message is created and passed to the external_mail() function (defined in scenario_interapplication_mail.php).
1:if (isset($_POST['action']) && $_POST['action'] == 'Register') {
2:// Create a welcome message
3: $subject = 'Welcome, ' . $_POST['name'] . ' - Account registration confirmation';
4: $message = '
5: <html>
6: <head>
7: <title>Welcome, ' . $_POST['name'] . ' - Account registration confirmation</title>
8: </head>
9: <body>
10: <h1>Welcome, ' . $_POST['name'] . ' - Account registration confirmation</h1>
11: <p>As a reference, please find your registration details below.</p>
12: <table>
13: <tr>
14: <th>Your name:</th>
15: <td>' . $_POST['name'] . '</td>
16: </tr>
17: <tr>
18: <th>Your e-mail address:</th>
19: <td>' . $_POST['email'] . '</td>
20: </tr>
21: <tr>
22: <th>Your password:</th>
23: <td>' . $_POST['password'] . '</td>
24: </tr>
25: </table>
26: </body>
27: </html>
28: ';
29:30:// Mail it
31: $registered = external_mail($_POST['email'], $subject, $message);
32: }
Let's have a look into the external_mail() function… The external_mail() function has the same signature as PHP's built-in mail() function, (See the PHP mail function - http://php.net/manual/en/function.mail.php) which should make transitioning from one to another very straightforward. It accepts a recipient, a subject, a body, optionally some additional headers and additional parameters for the e-mail sender.
1:function external_mail($to, $subject, $message, $additional_headers = null, $additional_parameters = null) {
2:// Create storage clients
3: $blobClient = createBlobClient();
4: $queueClient = createQueueClient();
5:6:// Generate a unique identifier for the mail message
7: $mailIdentifier = uniqid('', true);
8:9:// Create an array containing all parameters
10: $mailMessage = (object)array(
11:'to' => $to,
12:'subject' => $subject,
13:'message' => $message,
14:'additional_headers' => $additional_headers,
15:'additional_parameters' => $additional_parameters
16: );
17:18: try {
19:// Store the e-mail in blob storage
20: $blobClient->putBlobData('mail', $mailIdentifier, serialize($mailMessage));
21:22:// Add a message to the queue
23: $queueClient->putMessage('mail', $mailIdentifier);
24: } catch (Exception $ex) {
25:returnfalse;
26: }
27:28:// Success!
29:returntrue;
30: }
As a first step, a blob storage client and queue storage client are created. This has been outsourced to a createBlobClient() and createQueueClient() function doing the heavy lifting of connecting to storage and ensuring the necessary blob containers and queues exist. Next, a unique identifier for the message is generated using PHP's built-in uniqid() function. (See the PHP uniqid function - http://php.net/manual/en/uniqid) This identifier will be used to identify the message to be sent:
1: $mailIdentifier = uniqid('', true);
2:3://The message is then copied into an array which is cast to an object for easier access later on:
4: $mailMessage = (object)array(
5:'to' => $to,
6:'subject' => $subject,
7:'message' => $message,
8:'additional_headers' => $additional_headers,
9:'additional_parameters' => $additional_parameters
10: );
The message is then serialized into blob storage by using the putBlobData() function. After this, the message is also referenced in the queue using the putMessage() function. This process is wrapped inside a try/catch block, to ensure the function can return a false result if something fails along the way. Of course, it would be better to also log the exception if it occurs. Note that the order of action here is important: the message is first uploaded into a blob and only after that added to the queue. This is to ensure the actual message contents exist once another process processes the queue.
Sending out e-mail
The back-end process will run as a CLI application. It polls the queue using a backoff mechanism and sends out the e-mail stored in the queue service and blob storage whenever a message is present in the queue. After starting the script from the command-line (php scenario_interappliation_process.php), an infinite loop will be launched. The code for this loop can be found in the process_external_mail() function (defined in scenario_interapplication_mail.php).
Here is an overview of the complete function:
1:function process_external_mail() {
2:// Never stop...
3: set_time_limit(0);
4:5:// Create storage clients
6: $blobClient = createBlobClient();
7: $queueClient = createQueueClient();
8:9:// Backoff variables
10: $backOff = 0;
11: $backOffIncrement = 1;
12: $backOffMaximum = 30;
13:14:// Start processing
15:while(true) {
16: log_console("Checking for new messages...");
17:18:// Check for messages
19: $messages = $queueClient->getMessages('mail', 2);
20:21:if (count($messages) > 0) {
22: log_console("Found " . count($messages) . " messages to send.");
23:// Reset backoff value
24: $backOff = 0;
25: log_console("Backoff value has been reset.");
26:27:// Process messages
28:foreach ($messages as $message) {
29: try {
30: log_console("Sending message" . $message->MessageText . "...");
31:// Retrieve message from blob store
32: $messageData = unserialize($blobClient->getBlobData('mail',
33: $message->MessageText));
34:// Send mail
35: $messageSent = mail($messageData->to,
36: $messageData->subject,
37: $messageData->message,
38: $messageData->additional_headers,
39: $messageData->additional_parameters);
40:41:// Message sent?
42:if ($messageSent) {
43:// Delete it from the queue
44: $queueClient->deleteMessage('mail', $message);
45:46: log_console("Sent message" . $message->MessageText . ".");
47: } else {
48:// Failed sending message...
49: log_console("Failed sending message " . $message->MessageText . ".");
50:// If the message DequeueCount is > 10, move the message to a
51:// dead letter queue for investigation.
52:if ($message->DequeueCount > 10) {
53: log_console("Moving message " . $message->MessageText . " to dead letter queue.");
54: $queueClient->putMessage('deadmail', $message->MessageText);
55: $queueClient->deleteMessage('mail', $message);
56: }
57: }
58: } catch (Exception $ex) {
59: log_console($ex->getMessage());
60: }
61: }
62: }
63:64:// No messages found? Back off...
65:if (count($messages) == 0) {
66:if ($backOff < $backOffMaximum) {
67: $backOff += $backOffIncrement;
68: }
69: log_console("No messages found. Backing off for {$backOff}s.");
70: sleep($backOff);
71: }
72: }
73: }
Before diving into the actual queue processing, notice the control loop:
1:function process_external_mail() {
2:// Never stop...
3: set_time_limit(0);
4:// ...
5:6:// Backoff variables
7: $backOff = 0;
8: $backOffIncrement = 1;
9: $backOffMaximum = 30;
10:11:// Start processing
12:while(true) {
13:// ...
14:15:// No messages found? Back off...
16:17:if (count($messages) == 0) {
18:if ($backOff < $backOffMaximum) {
19: $backOff += $backOffIncrement;
20: }
21: sleep($backOff);
22: }
23: }
24: }
The polling script uses a backoff technique. If the script would poll the queue a few times every second and no messages are present, a lot of unnecessary queue transactions are performed (and billed!). To overcome this problem, the script polls the queue. If a message is present, the script processes it. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds. Next loop 3 seconds, four seconds, … up to a maximum backoff limit of 30 seconds for this script. If messages are present at a given time, this value is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm.
Within the control loop, the mail queue is checked for new messages. 2 messages are fetched at a time, but this number can be increased to a maximum of 32 messages that are fetched at once.
$messages = $queueClient->getMessages('mail', 2);
The message itself is processed and wrapped within a try/catch block. This has been done for two reasons: all Windows Azure SDK operations will throw an exception if something goes wrong. By catching exceptions the script is ensured to keep running forever until stopped. Otherwise, an exception would end the script. Next, the try/catch block has not been put around every separate storage transaction because every message should be processed as a whole: if one operation fails, the complete operation should fail.
Sending out the actual e-mail is done by passing the message contents into PHP's built-in mail() function.
1:// Retrieve message from blob store
2: $messageData = unserialize($blobClient->getBlobData('mail', $message->MessageText));
3:4:// Send mail
5: $messageSent = mail($messageData->to,
6: $messageData->subject,
7: $messageData->message,
8: $messageData->additional_headers,
9: $messageData->additional_parameters);
If sending the message succeeds, the queue message is deleted. If it fails, it is not deleted. This ensures that the message will be processed again later on.
1:// Message sent?
2:if ($messageSent) {
3:// Delete it from the queue
4: $queueClient->deleteMessage('mail', $message);
5: }
We don't want a message to block the complete script if it fails sending several times. When sending the message has failed for several times (in this script 10 times), the message is transferred to another queue, named deadmail, which allows the message to be inspected in another queue. The number of times a message has been dequeued for processing is noted in the queue message's DequeueCount property.
1:// If the message DequeueCount is > 10, move the message to a
2:// dead letter queue for investigation.
3:if ($message->DequeueCount > 10) {
4: log_console("Moving message " . $message->MessageText . " to dead letter queue.");
5: $queueClient->putMessage('deadmail', $message->MessageText);
6: $queueClient->deleteMessage('mail', $message);
7: }
