Peter in the Azure sky

Service Bus Queues

Azure Service Bus queues offer asynchronous messaging as a (PaaS) service. This means that someone can put messages on a queue and someone can consume them. Wouldn’t be much value from the service otherwise but it is important to understand that we are only talking about temporary message storing. For permanent storage you should not use queues. They can handle great scale compared to traditional synchronous requests. For instance you may not need the machine park for serving the maximum number of simultaneous requests but you may allow messages to be stacked up in the queue and processed during time as long as you can handle them over-time and the queue does not fill-up.

Cross platform (almost)

Messages are unstructured

One of the most difficult thing for people to comprehend with service bus messages are that the messages are not forcibly structured. If I have access I can post messages with nonsense data and I will succeed as there is no XSD format validation or similar when you post the messages.

It is therefor important to understand  that you may get data quality related issues when you try to interpret the messages.

Dealing with currupt messages

As described above, many of Microsofts Azure services the queueing components do not support schemed messages. This means that you cannot enforce that all messages in the queues follow a certain format. You have to deal with this when parsing the messages. There is built-in functionality that after (default) 10 tries  to consume a message, the message gets automatically put in the dead messages queue. Note: This value can be set/changed both in the portal and through code.

Security/Shared Access Secret Token

TCP vs HTTP Binding vs AMQP

Formats/Serialization

So when should you serialize as XML instead?

• You have an ancient version of your integration platform and don’t plan to upgrade (mind the expiration of support though, extended support is expensive and Microsoft is clearer and clearer propagating that they support the current version and one older version of each of their platforms. The trend is also auto upgrade to the highest version so that people will run on updated software so it might be time to upgrade. So don’t get to far behind in your system software.
• The posting application is older and cannot serialize JSON. But can they in that case create a SAS Token?

Don’t put a bottleneck in front of the service bus

An alternative to putting an extra layer in in front of the service bus one could to add some of local API component closer to the sender/receiver instead fronting the service bus with a commonly front to the service bus. You could also use your integration engine such as BizTalk to send/receive messages from/to the service bus from/to a source/destination that cannot speak directly with the service bus.

What to think about?

Shared access keys

I mentioned earlier that the shared access tokens are really safe. So that unless you have the secret key you cannot exploit the service bus. However if you have the key you can do whatever the access key has been configured to do. This fact dictates two important security measures you should take make it more difficult to exploit:

1. Assign minimal rights to resources (read more about this in section Set up Minimum Access/Partner rights).
2. Rotate the access keys. This means that you should periodically change the access key for the partner. Read up on how to do this in this section Shared Access Key rotation.

 Set up Service Bus for Minimum Access

• First of all I would recommend removing the default RootManageAccessKey on the namespace and create a new account for managing your service bus. This would make it more difficult to gain access over your namespace as the offender would already know the SharedAccessKey and only have to hack your secret assuming he/she knows your namespace. So set up another manage account (different key) for managing the Service bus. Maybe I am a bit paranoid here-  but it is a bit like not naming your computer administrator account Administrator.

  

• Set up access on resources rather than namespace. This means that you in a common send/receive scenario would need to assign the partner (send) rights on a request queue (incoming requests) and (listen) rights on the response/receive queue. This (by default) forces the partner to create different tokens for sending and receiving messages (or connection strings that you probably call it when using the .Net API). If the partner has access to many queues and you have set up password rotation you can get into an administrative nightmare and may cause the partner some headaches in setting the right key to the right queue. I will discuss how I handled this problem later in the post for minimum access/single access key on multiple entities.
• Set up separate shared access keys for all(each) partners. This enables you to remove access to a certain partner without affecting others, it does not have to be company it may very be company/actor (like a system). You can also better monitor activity. For example if multiple partners shall send messages to the same queue then don’t give them the same key/secret for posting messages.
• If you have an integration engine that is going to consume messages, like BizTalk, you could perhaps use a key/secret placed on the namespace that BizTalk should use to simplify management. But that is basically the only scenario that I recommend that you add keys and secrets on the entire namespace.

Shared access key rotation

In code that could look like this… (download sourcecode if you want a full running sample). The sample below accepts the name of a Shared Access Policy and swaps the keys. It will also do it for all SharedAccess policies with the same (in my case partner)name so if you have a partner sending and receiving on different queues this sample will change both to the same key.

/// <summary>
/// Moves the primary key to the secondary key and sets a new primary key for the partner
/// </summary>
/// <param name=”partnerName”>The partnername to replace rights for</param>
/// <param name=”newPrimaryKey”>return value the new primary key</param>
/// <param name=”newSecondaryKey”>retur value for the new secondary key</param>
/// <returns></returns>
internal bool ChangeAccessKeys(string partnerName, out string newPrimaryKey, out string newSecondaryKey)
{
newPrimaryKey = “”;
newSecondaryKey = “”;
NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(string.Format(SB_CONNECTIONSTRING_TEMPLATE, SBNameSpace, ManagePolicy, ManagePolicyAccessKey));
string newPrimary = SharedAccessAuthorizationRule.GenerateRandomKey();   //Randomize new key to use for all instances
var allQueues = namespaceManager.GetQueues().ToList(); //Fetch all queues from namespace
foreach (var item in allQueues)
{
//For selected queue see if there is a SharedAccessKEy for the partner
var rule = item.Authorization.Where(r => r.ClaimType == “SharedAccessKey”
&& r.KeyName.Equals(partnerName, StringComparison.InvariantCultureIgnoreCase)).FirstOrDefault();
if (rule != null)  //if was old rule then update it
{
SharedAccessAuthorizationRule sasRule = (SharedAccessAuthorizationRule)rule;
newSecondaryKey = sasRule.PrimaryKey;   //Just the out variable
sasRule.SecondaryKey = sasRule.PrimaryKey;  //Set secondary key to primary for this queue and “user”
sasRule.PrimaryKey = newPrimary;  //Set the new primary key
namespaceManager.UpdateQueue(item);  //Update Queue}
}
//No errors assume OK password change
newPrimaryKey = newPrimary;
return true;
}

Note that this (in a big scenario) would probably be done by having a scheduled web job, logic app or Azure function doing this on predefined intervals. With logic apps one could easily also imagine sending the new access secrets directly to a customer through SMS or a secure email service.. Future post perhaps….

At least once vs at most once – Message consumption pattern

Another thing you should consider is your consumption pattern. The following scenario could absolutely happen: You lock and read a message from a queue, before you can commit the message (DELETE in REST) the message is returned to the queue (by error or by plan) and another thread picks up the message again. The second process of the message does not know the state of the previous processor and how far it came and if the changes are rolled back or committed. While there are ways to reduce this possibility the simplest alternatives being longer lock time or renewing the lock in your code you cannot be 100% sure that your message is not consumed more than once in this scenario. And while WebJobs automatically renew the lock as the program runs – the program may still fail/hang and if you are unlucky it fails after you have committed your changes (even if you use transactions). So you should consider what is worse of the following two alternatives.

• The message is consumed more than once – not good in your accounting or bank system for example. Here you may want to process the message at-most-once but write to a log if it fails to repair from the logs. If you catch an error you may want to move the message to another queue or the DeadMessage queue instead of letting it be consumed again so you can take action on it but to risk crediting a bank account multiple times is not OK, it is better to raise an exception.
• It does not matter much if the message is consumed multiple times. While it is not fun with multi-posts to your timeline in Facebook it may not matter so much in the long run. You could remove the extra post quite easily or possibly even detect it by looking at identical posts from the last 15 minutes. This is also an OK scenario if you have some sort of logging, GPS tracking or fire alarm detection system where the potential loss of data is worse than duplicates.

So depending on your desired consumption pattern you would process messages differently.

At-most once
API

queueClient.Mode = ReceiveMode.ReceiveAndDelete;
message = queueClient.Receive(); //Directly removes the message from the queue
//Do your stuff.
//If fails you must handle it if you want to retry like put message somewhere else

REST

DELETE https://your-namespace.servicebus.windows.net/yourqueue/messages/head?timeout=60 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace&sig=Fg8yUyR4MOmXfHfj55f5hY4jGb8x2Yc%2b3%2fULKZYxKZk%3d&se=1404256819&skn=RootManageSharedAccessKey
Host: your-namespace.servicebus.windows.net
Content-Length: 0

At-least once
API

queueClient.Mode = ReceiveMode.PeekLock;  //Default
message = queueClient.Receive();
//Do your stuff.
message.Complete();   //Removes the message from the queue

REST

POST https://your-namespace.servicebus.windows.net/yourqueue/messages/head?timeout=60 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace&sig=Fg8yUyR4MOmXfHfj55f5hY4jGb8x2Yc%2b3%2fULKZYxKZk%3d&se=1404256819&skn=RootManageSharedAccessKey
Host: your-namespace.servicebus.windows.net
Content-Length: 0
//Do your stuff and use the message id + lock key from the broker properties to commit the message when successful, You could also use the location property for the DELETE
DELETE https://your-namespace.servicebus.windows.net/yourqueue/messages/31907572-1647-43c3-8741-631acd554d6f/7da9cfd5-40d5-4bb1-8d64-ec5a52e1c547?timeout=60 HTTP/1.1
Authorization: SharedAccessSignature sr=rukochbay&sig=rg9iGsK0ZyYlvhIqyH5IS5tqmeb08h8FstjHLPj3%2f8g%3d&se=1404265946&skn=RootManageSharedAccessKey
Host: your-namespace.servicebus.windows.net
Content-Length: 0

 

Handling rights on multiple resources

All operations in the service bus can be programmed against and also things as creating queues, access keys, changing secrets etc. As you probably want this to be handled without having someone going into the portal to add 10 separate queues and assign the new partner access to old resources you probably want to build an administrative user interface. And I am not talking about the service bus explorer but a more customized solution. I quickly discovered the need for this when implementing Azure Service bus at one of my customers.

But back to the problem first. So you have a partner that will post changes to 10 of your queues, and will receive messages in 15 (Response) service bus queues. And you don’t want them to have 25 access secrets for accessing each but still utilize minimal access rights.

What I did was to add the same access key to several resources by a little web app (similar code in downloadable in here) where I could add new partners, select which of the existing queues they should access with what right and add new queues.

If you add Shared Access Policies through the portal they will get separate access secrets to all queues which is very secure but could be difficult to maintain, the portal  assigns the same secrets to all the queues that the partner should be able to access. By having a tool like this you could also set up your other queue settings the way you want to without being a accustomed to and trained user in the portal. You can download a working samle of this related to this post (see beginning of post). Still separate acces keys offer better security but more problems when the keys needs to be updated.

Send Multiple messages in one go (Batch Sends)

If you are going to send multiple messages you can use the batch send functionality to send multiple messages, like for example CreateCustomer, CreateOrder that will happen simultaneous. You can send multiple messages in a batch which is more efficient than sending them individually. Once the service bus receives the messages it will split them to separate messages thereso it is very userfull if you can find a use-case for it.

Send multiple through API

QueueClient queueClient = QueueClient.CreateFromConnectionString(ConfigurationManager.AppSettings[“Microsoft.ServiceBus.ConnectionString”], “TestQueue”);List<BrokeredMessage> messageList = new List<BrokeredMessage>();

Microsoft.ServiceBus.Messaging.BrokeredMessage mes1 = new BrokeredMessage(“Msg 1”);
mes1.Properties.Add(“CustomerNo”, “1”);
messageList.Add(mes1);

Microsoft.ServiceBus.Messaging.BrokeredMessage mes2 = new BrokeredMessage(“Msg 2”);
mes2.Properties.Add(“CustomerNo”, “2”);
messageList.Add(mes2);

queueClient.SendBatch(messageList);

Send multiple through REST API

POST https://your-namespace.servicebus.windows.net/TestQueue/messages?timeout=60 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace&sig=ASADADADADASASASADADAS&se=1404256819&skn=Sender
Content-Type: application/vnd.microsoft.servicebus.json
Host: your-namespace.servicebus.windows.net
Content-Length: 18
Expect: 100-continue[
{
“Body”:”Msg 1″,
“BrokerProperties”:{“Label”:”M1″},
“UserProperties”:{“CustomerNo”:”1″}
},
{
“Body”:”Msg 2″,
“BrokerProperties”:{“Label”:”M2″},
“UserProperties”:{“CustomerNo”:”2″}
}
]

 

Note that Content-Type shuld be set to application/vnd.microsoft.servicebus.json

Important: As far as I can find out the old limitations on the Batch POST that it cannot be more than 256k in Size is still valid (should any reader have any other information then please make a comment and send the link). So you cannot send huge batches.

BizTalk / Partitions

If you intend to use BizTalk 2013 (and higher) as consumer/provider of messages through the SB-Messaging Adapter you must make sure that the queues are NOT set up as partitioned queues. Read the entire post if wrote earlier about this here.

Sidenote:  Partitioned queues simplifies having multiple consumers of messages and higher overall throughput. You can also set up rules for prioritized messages. Read more about partitions in the Azure Service Bus documentation.

Broadcasting changes with Topics

The above scenario works best with partner to partner communications like someone posting an update to be handled somewhere else or requests for specific data where a response is fetched to match the specific request and put in another queue.

If you have another scenario such as people subscribing to changes in the customer or article databases you probably want to use topics instead. In that scenario you can put one message on a “queue” that is then possible to received by multiple receivers (for example systems). Each of these receivers can consume the message separately so there is no competitiveness between subscribers. Another great feature with topics is that you can add logic to the subscribers such as individual subscribers only want swedish orders, another wants only “Big” orders (over € 10000).

Synchronous integration with (most likely) on premise resources

The previously described scenarios (queues and topics) are all aimed to deliver asynchronous message-based communications. What if you need to access on premise resources synchronously?

Before continuing the discussion about Azure Service Bus, there are other alternatives for on premise communication between Azure (the cloud) and on premise resources including Site-Site VNets, Point to Site VNet, Onpremise Data Gateway (I beleive they work with Service Bus Relays internally but I am not absolutely sure), Logic Apps (some adapters can be set up to connect to on premise databases with some simple steps and to SOAP services with some small adjustments as well). But this discussion was about the service bus so I will discuss the method on how to work manually with the servicebus instead.

Service Bus relays enable you to create WCF services (REST/SOAP) that run on premise inside your corporate firewall and therefore likely are not subject to that many regulations on what they can access. They could access files, databases, etc (you name it) just like any other on premise application. The big difference is that they use also go out to Azure (though TCP or HTTP(s)) and register themselves in the Service Bus (if they have the correct rights set up of course). You can then assign B2B partners to access secrets and allow them to call through service bus authorization) right in to your on premise hosted services. The benefits of course that the callers does not know the details of your on premise implementation but only how to connect to the service bus service.

You pay for the number of hours a relay service is online * the number of instances. So the more instances -> the more it will cost.

High availability for Service Bus Relays

You should set up your relays with the appropriate failover matching your needs. A B2B communication may be considered online as long as the systems can communicate with the service bus queue (for queue based messaging) but for relays you can’t really say the same if you get a 404 because the services are unreachable. So when designing B2B solution based on the Service Bus Relays you should do the following:
1. Add a retry functionality at the sender (caller) easilly done if you use BizTalk or other integration software for calling the relay)
2. Have multiple instances (ideally 3 or more if performance does not dictate more) of the Relay Service (WCF service on premise). If you have two data centres with different internet routing it could be wise to put one instance at the other location.

Avoid long-running relay services

As always when it comes to synchronous calls you should avoid long-running processes. For report creation and other time consuming services put the request on a queue handler (SB queue, MSMQ queue, Windows Service Bus) and have a background job creating the reports. You could then return “Report request submitted” directly and can have your service handling other incoming requests instead. This is not specific to Service Bus Relays but is a good architecture overall. Consider that the first version of Azure had Web Roles and Worker Roles. Web roles handled web requests and worker roles handled heavy/batch processing.