Server requests accumulating indefinately

ServiceStack Server Events
1

We use ServerEventsFeature in our application for sending notifications to clients. First day the application went live we found that IIS stopped working after some time with HTTP 503.2 since number of parallel requests exceeded max of 5000. Looking into IIS running requests list I found that all requests were for event-stream. Further checks showed that when client closes event source, the request on IIS continue to run, so even there were about 60 clients, IIS got stuck with 5000 requests.

The problem was fixed (bypassed?) by adding RemoveExpiredSubscriptions into response filter.

Is SSE supposed to be used in this way? I did not see mentioning of this in ServiceStack doc.
If not, how it is supposed to work?

2

Server Events has heartbeats enabled and automatically closes any expired connections, what does your ServerEventsFeature registration look like?

3

Plugins.Add(new ServerEventsFeature() { NotifyChannelOfSubscriptions = false });

4

The expired connections should be removed on each heartbeat, can you look at the HTTP traffic (using WebInspector or Fiddler) and check to make sure the heartbeat requests are successful?

5

My client is Web app, so it uses standard EventSource class.

Do I need to configure heartbeat on the client?

6

Well then that will be the issue, yeah it needs to send heartbeats - but they’re automatically configured in all the available ServerEvents clients so you don’t need to configure it manually.

For a plain JavaScript Web App you can use the ss-utils JavaScript client which also makes it much easier to handle and integrate with ServiceStack’s Server Events.

For TypeScript Web Apps you can use the TypeScript ServerEventsClient.

7

I’ve read the article but it does not mention heartbeats.

Can you clarify whether my approach (plain EventSource + RemoveExpiredSubscriptions) will work OK?

8

If you use either any of the JavaScript clients heartbeats are automatically configured so you don’t have to worry about configuring it yourself - i.e. they’re a transparent implementation detail.

No plain EventSource is not sufficient, you need a client that also sends heartbeats (i.e. any of the clients provided) so the server knows to keep the connection alive and can perform the necessary house-keeping.

9

Do I understand correct that I need to add to work with ss client while the my JS remains intact?

10

I don’t understand the question. If it’s do you need to use an SS Server Events Client like I’ve listed above? Yes, you should.

Technically you don’t need to if you manually handle sending heartbeats and calling unregister URL when disconnecting, retry on failed connections/heartbeat failures etc. The provided clients transparently handles these book keeping details and makes handling Server Events much easier to work with. Whist not using the clients is what’s causing the undefined behavior you’re seeing which need to be manually configured to handle the book keeping tasks the existing Server Events client are already configured to handle.

11

I meant that adding tag “script src=”/js/ss-utils.js" will use SS client instead of standard EventSource.

12

That just references the JavaScript library so you can access it, the documentation for how to use it is in: http://docs.servicestack.net/javascript-server-events-client

Which starts with:

var source = new EventSource(
    '/event-stream?channel=channel&t=' + new Date().getTime());

$(source).handleServerEvents({ 
  //...
});
13

We currently use AddEventListener. Will it still work with ss-utils.js or I ahve to use $(source).handleServerEvents?

How does heartbeat look? I’d like to see it in browser’s network trace.

14

I tried to use ss-utils but found that we need to change significantly our code: remove event source polyfill, start using selectors (until now we send empty selector, but ss-util fails on it).

If we stay with our implementation (plain EventSource), why do we need heartbeat? How can we add it?

15

Please read through my answers very carefully so I’m not having to keep repeating myself.

You absolutely need a Server Events client that sends heartbeats, Server Events wont function properly if it doesn’t send heartbeats. This entire thread is because you’re using a client that doesn’t send heartbeats. You can send heartbeats by using any of the clients I’ve linked to above. In addition to sending heartbeats each clients automatically connect/retry when it detects an event-stream no longer exists. If you don’t use any of the existing clients you’ll be using an unsupported configuration that will continue to have issues until it’s configured to send heartbeats. You can look at the source code of ss-utils to see the code we use to configure and send periodic heartbeats.

16

Thanks @mythz, we will rework our implementation using ss-utils.

17

Hi @mythz.
We changed our client implementation and using now ss-utils.js.
Mainly it work but we still have a problem with lost notifications after IISRESET. Client does not reconnect automatically to the channel and notifications sent after IISEREST do not reach clients until they relogin.

Is there something that we can do with this?

18

That’s not the behavior we’re seeing which auto reconnects after receiving a failed heartbeat. The clients aren’t going to receive any messages whilst they’re reconnecting (i.e. when they’re not connected), but they should auto reconnect. You can test this on the Chat Server Events Example App.

Make sure your heartbeats are working and when you do an IIS Reset the clients are sending/receiving a failed heartbeat.

19

I checked that GET for event-stream gets HTTP 503 at the time of IISRESET.
GET is not reissued by client and heartbeats are not sent anymore after this.

Do I need to do something from client side to renew the connection and heartbeats?

20

No the defaults should work, are there any JavaScript errors in WebInspector?

The /js/ss-utils.js should be easy to debug since it’s unminified, are you able to tell where it gets stuck?