SSE Client in a SS Service

[jezzsantos]

Hey Mythz,

I’d like to discuss a design issue with having an SSE client running from within a ServiceStack service.

I cannot seem to wrap my head around the lifecycle part of the SSE client in the lifecycle of my SS service. I’ve not had to think about keeping anything else alive with my SS service until now. But the SSE client needs to be alive from the moment the service starts.

Basically, I’ll need a (single) instance of an SSE client to be alive for as long as my service is alive, so that it instruct my service to do stuff when a SSE event comes through from some SSE server elsewhere.

So initially, I thought creating the SSE client in my AppHost.Configure() method was the right way to go.

I have subsequently discovered that if I do that, and hold the SSE client instance in my AppHost, then if the SSE client throws an exception (say because it lost connection or something with SSE Server) than it seems to throw and bring down my SS service. Presumably, killing my AppHost instance?

This is what seems to happen. I can detect that my SSEClient.OnException delegate is called and shortly after my service dies. I am assuming they are related, since we had not had problems with the service dying before we added this SSE client.

If this is by design, then how would you manage the lifecycle of an SSE client to be alive as long as the SS service without bringing the service down when the SSE client throws an exception?

[mythz]

The client should just be like any other dependency, if you want to keep an instance around you should assign the instance to a field in AppHost or register the singleton instance in the Container so there’s a reference to it.

But this shouldn’t be happening, if the SSE client connection is broken it should be automatically be retrying to connect. Do you have a StackTrace showing what’s causing this or a small repro that exhibits this behavior?

If it’s bringing down the ASP.NET App Domain down I’m assuming it’s due to an UnobservedTaskException which you can handle by registering a handler in App_Start, e.g:

protected void Application_Start(object sender, EventArgs e)
{
    new AppHost().Init();

    TaskScheduler.UnobservedTaskException += (source, args) => args.SetObserved();
}

This should prevent unhandled async exceptions from bringing down the Web App, you’ll also want to add some logging around this as well to see what’s causing this. There’s some more details about unhandled ASP.NET async Exceptions in this blog post.

[jezzsantos]

OK, this is what happens:

We have this code in the Global.asax.cs of our SS service. Neither of these breakpoints are ever hit!

When we debug the service, the debugger catches this unhandled exception:

Our (Azure) trace log looks like this:

In our AppHost we are calling this function to register our SSE client.

    internal class AppHost
    {
        private ServerEventsClient serverEventsClient;

        public override void Configure(Container container)
        {
            base.Configure(container);

    ... more registrations

            CreateServerEventsClient(container, container.Resolve<IClientConfiguration>());
        }

        private void CreateServerEventsClient(Container container, IClientConfiguration configuration)
        {
            var baseUrl = configuration.CribServicesApiBaseUrl;
            this.serverEventsClient = new ServerEventsClient(baseUrl, Channels.VerificationUpdate);
            this.serverEventsClient.RegisterReceiver<GlobalReceiver>();

            container.RegisterAutoWiredTypes(this.serverEventsClient.ReceiverTypes);
            this.serverEventsClient.Resolver = container;

            this.serverEventsClient.Start();
        }
    }

[jezzsantos]

I tried to debug ServiceStack.Client.dll for you to try and identify the exact line throwing the exception, but my debugger cannot load the symbols for this current version (v55) from the known symbols places.

[mythz]

I can’t repro any unhandled exception, I’ve got it running with the network off and it’s just retrying continually as expected. Are you sure your exception is an unhandled exception and it’s not just stopping on a normal exception? i.e. does it say anywhere it’s an unhandled exception?

I’m going to let it run with the network off for another 10 more minutes with continual failed heartbeats + retries and see if I can get it to repro an unhandled exception, will let you know if it reconnects or when the networks back or not.

[mythz]

I’ve jumped on the CI machine and published the latest v4.0.55 MyGet packages and symbols onto GitHub which you can get at: https://github.com/ServiceStack/Assets/tree/master/nuget/v4.0.55

[mythz]

Nope still can’t repro any unhandled exception, I’ve left my network disconnected for over 20 minutes and as soon as I reconnected the network again it automatically re-established the SSE connection to the remote http://chat.servicestack.net SSE Server and started receiving messages again. AFAICT it’s working as expected.

[mythz]

I’ve tried restarting the server a couple times and found one NRE which should be guarded against in this commit.

This change has just finished deploying on MyGet.

[jezzsantos]

Hey Mythz,

Great news! I still get the failure to connect with SSE server, but this time no exception is caught by the debugger. So I think you have fixed it!

AND, my service is not killed!

so thanks for all that. You must have killed the right NRE.

[mythz]

Ok great, these suckers can be hard to hunt down