ServerSentEvents with NotifyChannelOfSubscriptions set to false leaks requests?

ServiceStack Server Events
1

We don’t want the onLeave/onJoin messages to be sent to clients, so we’ve set NotifyChannelOfSubscriptions to false on ServerEventsFeature.

We’ve noticed that after some requests IIS would stop answering, and would only resume working after iisreset.

After some debugging, we’ve noticed that the requests for the /event-stream endpoint would remain running, even refreshing the browser/navigating to another page. We found that by taking a dump of the iis app pool process and looking for the HttpContext instances, and by looking at the performance counters for “current connections” (Get-Counter "\web service($appPoolName)\current connections" -SampleInterval 1 -MaxSamples ([int]::MaxValue)).
The performance counter would increase on each refresh until IIS would stop responding.

We’ve isolated this behaviour to the NotifyChannelOfSubscriptions flag on the plugin, which we had set to false.

This looks like a bug. Can you confirm?

2

I don’t see how this would have any effect, can you post your entire ServerEventsFeature registration with any non-default configuration that may have an effect on this.

3

I was also somewhat surprised, but I’ve reduced it all to:

    public override void Configure(Container container)
    {
        Plugins.Add(new ServerEventsFeature
        {
            NotifyChannelOfSubscriptions = false,
        });

        container.Register<ICacheClient>(new MemoryCacheClient());
    }

This is enough to trigger the problem. Tested with 56 and 60.
Here’s a repo with a repro: https://github.com/brunomlopes/servicestack-connleak-repro

  • Download, compile, set it up on IIS.
  • Hit the event-stream endpoint 12 times in a row, from the tenth or so it will stop answering.

I’ve used the following line on powershell to quickly hit the endpoint:

0..15 | % { curl.exe -m 1 "http://localhost/event-stream?channels=chan&t=1466451780049" }

After the 11 or 12th request (depending on whether it takes one or two seconds for the app pool to initialize), this will show:

curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received

And it would take an iisreset to get it to respond again.

4

This was painful to resolve and I haven’t yet determined the exact reason why it hangs just that it’s happening with ASP.NET Apps deployed locally on IIS, i.e. it doesn’t happen with IIS Express or self-hosts. I’d say this a pretty severe bug with IIS, as just 10 open connections are enough to make it hang and stop any more requests from reaching ServiceStack, where restarting the Web App, App Domain or IIS Website can’t fix it. It lets you stop the AppDomain but refuses to start it, the only thing that works is a hard iisreset as you’ve noticed.

The fix to resolve this issue is that we need to write something to all channel subscribers so we get an immediate Exception for any subscriptions (i.e. long-running http requests) that no longer maintain an active tcp connection, so we can dispose of them immediately. Since it doesn’t have any effect if NotifyChannelOfSubscriptions = false we just write an \n which is enough to trigger an Exception for lost client connections.

This fix is available from v4.0.61 that’s now available on MyGet.

5

Hi Mythz,

This issue seems to still be there. I downloaded the same repo above, upgraded to latest 5.1.0.
and ran the snippet(below) x15.

0..15 | % { curl.exe -m 1 "http://localhost:8484/event-stream?channels=chan&t=1466451780049" }

and then started to see the following.

data: chan@cmd.onJoin {"userId":"-10","isAuthenticated":"false","displayName"
28665013356","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Ass

curl: (28) Operation timed out after 1000 milliseconds with 811 bytes receive
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1015 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1016 milliseconds with 0 bytes received

consequently making IIS totally unresponsive and requiring issreset for it to come back to life.

6

Update to v5.1.1 on MyGet.