TLDR
If immediate action is not a hard requirement, then you could tackle service unavailability by implementing retry-logic when consuming endpoints which do not guarantee availability.
Context
Those who know me well, know how much I like Azure Service Bus and its capability of decoupling applications, which comes in handy when designing microservices. However, choosing the correct tier can be tricky, given the huge differences in the pricing levels. Leaving aside the Basic (Classic aka Old) tier which should only be used for test & development purposes, we have to chose between the Standard tier and the Premium tier. The Standard tier is far cheaper (although it can get expensive if ones sends large numbers of messages) but can be unstable from time to time which means that the Service Bus might become inaccessible and unavailable to receive messages. The Premium tier on the other hand, has fantastic availability is far more trust worthy but costs almost 100 times more than the Standard tier.
So the dilemma is, do I value availability (and the other less significant extras) higher enough to pay that price? If yes, then you might as well stop reading my blog post right now. If availability is important but you could live with a small latency now and then, then I’ve got good news for you!
Using Durable Functions, you can instruct the Orchestrator-function to try until the message is sent, regardless if it succeeds on the first try or if it takes 100 tries, it will keep on trying. Take a look at the following code:
var options = TaskOptions.FromRetryPolicy(new RetryPolicy(maxNumberOfAttempts: 20, firstRetryInterval: TimeSpan.FromSeconds(5), backoffCoefficient: 3, maxRetryInterval: TimeSpan.FromMinutes(1)));
await context.CallActivityAsync<ActivityFunctionResponse>(nameof(ServiceBus.SendMessageToServiceBus), args, options);The context object in the code above is my OrchestrationTrigger and its namespace and type is Microsoft.DurableTask.TaskOrchestrationContext. Keep in mind that my code is written for Isolated Durable Functions, on dotnet 7. The args object contains the parameters I wish to send to the Activity Function.
Now what does my retry logic say? They that the orchestrator should call the activity function (which in my case sends a message to the service bus and throws exception if it fails) maximum 20 times, the first retry will happen after 5 seconds, the wait interval will triple for each retry (so 5, 15, 45 … etc seconds) and will get as high as 1 minute. In other words, the orchestrator will be calling the function for about 20 minutes. I have been running this code for months now, and have never missed sending a message, although I sometimes have to wait for the Service Bus (which runs on the cheap Standard tier) to …feel better.
