Hi, I need to run a long-running process (let’s say upto an hour) on an azure website which is based on ServiceStack. What is the best pattern for long-running durable processes that will work within Azure’s threading model?
I want the INSTIGATOR client to start the job off and be notified of progress (even if the client leaves the progress page and then returns - progress update should be accurate and updated in real time. If a new client (different user/machine etc) navigates to the progress page, they should also be notified. There are a couple of options here - any input would be appreciated.
I already use Azure Redis for Caching. The options as I see it are:
1. Instigator calls service which creates an appropriate row in a table representing the Job, and also a Azure Queue item which triggers WebJob within Azure. The WebJob code updates the Job row in the table with progress, meanwhile the client is polling a service that returns the current progress. Doesn’t fill me with joy this one.
2. Use RedisMQ and Server Sent Events to do this. Instigator submits request to start job and at same time opens EventSource to listen for SSEs. As progress is updated, an SSE is sent out. SS example seems to suggest this is possible, but don’t know the threading implications. I guess the question here is would this work within an Azure website and Azure Redis? or do you need some other runtime which are executing the long-running stuff? This is my preferred approach if it’s feasible as it would mean that all clients would be updated of progress without polling.
My personal pref is #2, nice thing about SSE is that it’s just a long-lived HTTP connection so should work anywhere HTTP does. Azure Redis Cache is full-featured so that should be fine. RedisMQ is basically just background threads running in ASP.NET app on App_Startup. I didn’t think there was a different threading model for Azure Websites, AFAIK it’s just a normal ASP.NET web app? (i.e. Win2008/IIS) so background threads should be ok.
Graham Laidler:
Ok great - just wanted to check there wasn’t any dependency on another background runtime - excellent. Many thanks.
Marc-Antoine Latour:
Keep in mind the danger of using background thread in asp.net
I strongly recommend that your long running task runner implement the IRegisteredObject interface to properly stop your long running operations since the AppDomain could be shutdown by IIS during a AppPool recycling…
And also un handle exception in background thread will kill the AppDomain and doing so more than a couple of time for a certain period will result by having IIS not starting your website anymore and client will start receiving a Service Unavailable response
A good read
http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx/
Sure, tho we were running RedisMQ in StackOverflow Careers for over a year without any exceptions bringing down the AppDomain. The one thing I will add is that if the website is not frequently hit it’s a good idea to add a periodic ping check (e.g. using Pingdom or Runscope) to keep the AppDomain up, since background threads themselves wont.
Graham Laidler:
Thanks all - Demis, did you do anything with IRegisteredObject when using RedisMQ previously?
Nope I didn’t register it with IRegisteredObject myself.