Server events and reverse proxy

Hello.
We have a problem while using ServiceStack.ServerEventsFeature inside a self hosted application beyond a reverse proxy server.
I am going to try explaining our scenario (simplifing).

We have a self hosted application powered by ServiceStack which exposes restful services to clients consisting in angular SPA applications.
One our customer requires a Single Sign On using the Windows authentication (NTLM or Kerberos) with a “trasparent user authentication”, so avoiding user credential input.
We chose to adopt Windows integrated authentication negotiated between user agent and IIS in order to do this job easy.
This choise invole configuring and using IIS as reverse proxy beyond the ServiceStack self hosted application.

So in this case:

  1. IIS authenticates requests recieved from clients (user agents) using its built in Windows integrated authentication without require user credential input and then forwards them to the ServiceStack self hosted application adding a custom header where logon user name is putting into;
  2. The ServiceStack self hosted application trusts all requests forwarded from IIS (in the real case we use also an addictional custom security protocol but is not relevant for the explaination) and managing sessions by ServiceStack.SessionFeature and using the forwarded logon username to manage profiling and authorizations logics.

First, in this scenario we caught that updateSubscriberUrl, heartbeatUrl and unRegisterUrl report the back end’s (the self hosted application’s) domain instead of the reverse proxy’s domain.
To fix it We have overridden these url in the OnConnect callback of the ServiceStack.ServerEventsFeature class.

Then We have been having another problem: while using a reverse proxy it seams that cannot create channels starting the event stream, or something like this.
We also compared the behavior detected using the reverse proxy (IIS) and excluding it, catching the following differences.

  • Using the reverse proxy the request “GET https://{reverse proxy}/event-stream?channels=void&t={timestamp} HTTP/1.1” does not returns “data: {channel}@cmd.onUpdate {…}” and “data: cmd.onHeartbeat {…}”, but only “data: cmd.onConnect {…}” and “data: void@cmd.onJoin {…}”.
  • Using the reverse proxy no request “POST http://{reverse proxy}/json/reply/UpdateEventSubscriber HTTP/1.1” was made.
  • Using the reverse proxy all request “POST http://{reverse proxy}/event-heartbeat?id=mI9ezDv6vRI3qEv0JtgM HTTP/1.1” return responses like this: “HTTP/1.1 404 Subscription ‘{id}’ does not exist”.

Using reverse proxy the only way to avoid the http 404 response is setting the configuration option “NotifyChannelOfSubscriptions” to false.

Where we are wrong?

Thank you in advance

So to start off, Server Events SSE just uses normal HTTP Requests, the only difference from a normal HTTP Request is that instead of sending short-lived requests, the /event-stream request maintains a long term persistent connection which results in the same behavior as a HTTP request that takes a very long time to complete. So all the normal rules & behaviors of normal HTTP Requests (e.g. Cookies) also apply to SSE requests because that’s all they are.

The primary issue with trying to run Server Events behind a proxy is to disable buffering.
https://docs.servicestack.net/server-events#troubleshooting

Because in order for the client to receive a notification sent from the server, it needs to be delivered immediately when written to the response OutputStream, otherwise when the server sends it it gets buffered by the proxy and only delivered when the response is flushed to the client.

The URL Rewriting is usually handled by the proxy, otherwise you can set Config.WebHostUrl or override GetBaseUrl() in your AppHost to override the URL ServiceStack uses.

The 404 is likely due to heartbeats not being delivered on time resulting in the server automatically unregistering the SSE connection, so when it finally arrives the subscription no longer exists. One thing to check for is that the cmd.onConnect event is immediately received in the client after a connection, which is what the client uses to setup a connection and periodic heartbeats.

Users subscribe to channels either in the original SSE /event-stream?channels=.. request or after they’re connected by sending an update request to /event-subscribers. If the issue is changing channels after they’ve already connected inspect the client HTTP Request Headers to make sure the request is being sent and the HTTP Request Headers on the server to make sure the request makes it to the server immediately (i.e without buffering). You should also be able determine this by inspecting the client headers to check that their request gets an immediate response from the server.

This just turns off sending channel notifications when other users in the channel leave or join, it wouldn’t fundamentally change anything other than sending less notifications.

Hello.
Thank you for the previous answers.
So, We made the following changes.

  • We disabled buffering as described in https://docs.servicestack.net/server-events#troubleshooting;
  • We fixed url reported in the event stream (for example http://{server}/event-heartbeat?id={subscription id}) overriding properly the method “ServiceStackHost.GetBaseUrl”;
  • We configured IIS Application Request Routing (the IIS module used to set IIS as reverse proxy) in order to avoid cache and buffering.
  • We turned on again the setting NotifyChannelOfSubscriptions.

After these changes we have done some test and at the end We continue to have the same kind of troubles.

We tested and compared (again) the following three scenarios.

  1. The ServiceStack application (self hosted) without reverse proxy beyound with a CredentialsAuthProvider, managing the authentication inside our self hosted.
  2. The ServiceStack application (self hosted) with the reverse proxy (IIS) beyound it and with a custom AuthProvider which authorizes everything bypassing the authentication (made by the reverse proxy); in this case the reverse proxy communicates user identities to the ServiceStack self hosted application where sessions have been managed by SessionFeature.
  3. The ServiceStack application (self hosted) without the reverse proxy (IIS) and with a custom AuthProvider which authorizes everything (where IsAuthorized method returns always true) bypassing the authentication (made by the reverse proxy); in this case used for test only, user identities are forced as if they have been recieved from the reverse proxy by the ServiceStack self hosted application where sessions have been managed by SessionFeature.

In the first and in the third scenario all works as expected.
In the second scenario it seams to have problems with sse subscriptions yet, probably for some kind of reason related to the reverse proxy.

Here We are reporting a part of the communication tracked by Edge developer tools in the first and in the second scenario.
The major difference found out is the lack of “cmd.onUpdate” in event stream and of http://{server}/json/reply/UpdateEventSubscriber POSTs in the scenario affected by troubles.

Could you help us to get back to the right way?
Have you any new idea about any undetected potential mistake we could had made watching the communication trace below?

Thank you again in advance.

Scenario 1

  1. First event stream.
    Request URL: http://{server}/event-stream?channels=void&t=1625215594720
    HTTP method: GET
    HTTP status: 200 / OK
    Request headers
    Accept: text/event-stream
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT, it; q=0.5
    Cache-Control: no-cache
    Connection: Keep-Alive
    Cookie: ss-id=pG7UAIsquFOxVn74JTxU; ss-pid=ObYcL0EFBbekMNHBw9OW; X-UAId=; ss-opt=temp
    Host: {server}
    Referer: http://{server}/html/it/main/index.html
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
    X-Requested-With: XMLHttpRequest
    Response headers
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Cache-Control: no-cache
    Content-Type: text/event-stream
    Server: Microsoft-HTTPAPI/2.0
    Transfer-Encoding: chunked
    Vary: Accept
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Event stream
    id: 1
    data: cmd.onConnect {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","id":"jBgzScprAJNHGgGnxtI0","unRegisterUrl":"http://{server}/event-unregister?id=jBgzScprAJNHGgGnxtI0","heartbeatUrl":"http://{server}/event-heartbeat?id=jBgzScprAJNHGgGnxtI0","updateSubscriberUrl":"http://{server}/event-subscribers","heartbeatIntervalMs":"30000","idleTimeoutMs":"60000"}

id: 2
data: void@cmd.onJoin {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"void"}

id: 3
data: ch_notifiche_main@cmd.onUpdate {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"ch_notifiche_main","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"ch_notifiche_main"}

id: 4
data: ch_notifiche_main@cmd.onUpdate {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"ch_notifiche_main","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"ch_notifiche_main"}

id: 5
data: ch_notifiche_main@cmd.onUpdate {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"ch_notifiche_main,ch_dashboard_main","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"ch_notifiche_main"}

id: 6
data: ch_dashboard_main@cmd.onUpdate {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"ch_notifiche_main,ch_dashboard_main","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"ch_dashboard_main"}

id: 7
data: cmd.onHeartbeat {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"ch_notifiche_main,ch_dashboard_main","createdAt":"1625215594783","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png"}

  1. First event subscriber update.
    Request URL: http://{server}/json/reply/UpdateEventSubscriber
    HTTP method: POST
    HTTP status: 200 / OK
    Request headers
    Accept: */*
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT, it; q=0.5
    Cache-Control: no-cache
    Connection: Keep-Alive
    Content-Length: 102
    content-type: application/json
    Cookie: ss-id=pG7UAIsquFOxVn74JTxU; ss-pid=ObYcL0EFBbekMNHBw9OW; X-UAId=; ss-opt=temp
    Host: {server}
    Origin: http://{server}
    Referer: http://{server}/html/it/main/
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
    Request body
    {"subscribeChannels":["ch_notifiche_main"],"unsubscribeChannels":["void"],"id":"jBgzScprAJNHGgGnxtI0"}
    Response headers
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Content-Type: application/json; charset=utf-8
    Server: Microsoft-HTTPAPI/2.0
    Transfer-Encoding: chunked
    Vary: Accept
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Response body
    {}

  2. First event subscription check (it seams to be a check).
    Request URL: http://{server}/json/reply/UpdateEventSubscriber
    HTTP method: GET
    HTTP status: 200 / OK

  3. Second event subscriber update.
    Request URL: http://{server}/json/reply/UpdateEventSubscriber
    HTTP method: POST
    HTTP status: 200 / OK
    Request headers
    Accept: */*
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT, it; q=0.5
    Cache-Control: no-cache
    Connection: Keep-Alive
    Content-Length: 122
    content-type: application/json
    Cookie: ss-id=pG7UAIsquFOxVn74JTxU; ss-pid=ObYcL0EFBbekMNHBw9OW; X-UAId=; ss-opt=temp
    Host: {server}
    Origin: http://{server}
    Referer: http://{server}/html/it/main/
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
    Request body
    {"subscribeChannels":["ch_notifiche_main","ch_dashboard_main"],"unsubscribeChannels":["void"],"id":"jBgzScprAJNHGgGnxtI0"}
    Response headers
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Content-Type: application/json; charset=utf-8
    Server: Microsoft-HTTPAPI/2.0
    Transfer-Encoding: chunked
    Vary: Accept
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Response body
    {}

  4. Second event subscription check (it seams to be a check).
    Request URL: http://{server}/json/reply/UpdateEventSubscriber
    HTTP method: GET
    HTTP status: 200 / OK

  5. First heartbeat.
    Request URL: http://{server}/event-heartbeat?id=jBgzScprAJNHGgGnxtI0
    HTTP method: POST
    HTTP status: 200 / OK
    Request headers
    Accept: */*
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT, it; q=0.5
    Cache-Control: no-cache
    Connection: Keep-Alive
    Content-Length: 0
    content-type: text/plain
    Cookie: ss-id=pG7UAIsquFOxVn74JTxU; ss-pid=ObYcL0EFBbekMNHBw9OW; X-UAId=; ss-opt=temp
    Host: {server}
    Origin: http://{server}
    Referer: http://{server}/html/it/main/
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
    Response headers
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Server: Microsoft-HTTPAPI/2.0
    Transfer-Encoding: chunked
    Vary: Accept
    X-Powered-By: ServiceStack/5,50 Net45/Windows

  6. Second event stream
    Request URL: http://{server}/event-stream?channels=void&t=1625215594720
    HTTP method: GET
    HTTP status: 200 / OK
    Request headers
    Accept: text/event-stream
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT, it; q=0.5
    Cache-Control: no-cache
    Connection: Keep-Alive
    Cookie: ss-id=pG7UAIsquFOxVn74JTxU; ss-pid=ObYcL0EFBbekMNHBw9OW; X-UAId=; ss-opt=temp
    Host: {server}
    Last-Event-ID: 7
    Referer: http://{server}/html/it/main/
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
    X-Requested-With: XMLHttpRequest
    Response headers
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Cache-Control: no-cache
    Content-Type: text/event-stream
    Server: Microsoft-HTTPAPI/2.0
    Transfer-Encoding: chunked
    Vary: Accept
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Event stream
    id: 1
    data: cmd.onConnect {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625215645244","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","id":"7TJcegfNvcD617bs4jDR","unRegisterUrl":"http://{server}/event-unregister?id=7TJcegfNvcD617bs4jDR","heartbeatUrl":"http://{server}/event-heartbeat?id=7TJcegfNvcD617bs4jDR","updateSubscriberUrl":"http://{server}/event-subscribers","heartbeatIntervalMs":"30000","idleTimeoutMs":"60000"}

id: 2
data: void@cmd.onJoin {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625215645244","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"void"}

id: 3
data: cmd.onHeartbeat {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625215645244","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png"}
`

Scenario 2

  1. First event stream
    Request URL: http://{server}/event-stream?channels=void&t=1625237024341
    HTTP method: GET
    HTTP status: 200 / OK
    Request headers
    Host: {server}
    Connection: keep-alive
    Pragma: no-cache
    Cache-Control: no-cache
    Accept: text/event-stream
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
    Referer: http://{server}/html/it/main/
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7
    Cookie: ss-opt=temp; X-UAId=; ss-id=sb0aBSSCZRSvraEFjJ6X; ss-pid=OYVgA2o3qOZFjbi5ipx7
    Response headers
    Cache-Control: no-cache
    Transfer-Encoding: chunked
    Content-Type: text/event-stream
    Vary: Accept
    Server: Microsoft-IIS/10.0
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Credentials: true
    X-Powered-By: ARR/3.0
    Persistent-Auth: true
    X-Powered-By: ASP.NET
    Event stream
    id: 1
    data: cmd.onConnect {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625237024260","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","id":"AKPtXBz4FlF79VwG26iM","unRegisterUrl":"http://{server}/event-unregister?id=AKPtXBz4FlF79VwG26iM","heartbeatUrl":"http://{server}/event-heartbeat?id=AKPtXBz4FlF79VwG26iM","updateSubscriberUrl":"http://{server}/event-subscribers","heartbeatIntervalMs":"30000","idleTimeoutMs":"60000"}

id: 2
data: void@cmd.onJoin {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625237024260","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"void"}

  1. Second event stream
    Request URL: http://{server}/event-stream?channels=void&t=1625237024341
    HTTP method: GET
    HTTP status: 200 / OK
    Request headers
    Host: {server}
    Connection: keep-alive
    Pragma: no-cache
    Cache-Control: no-cache
    Accept: text/event-stream
    Last-Event-ID: 2
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
    Referer: http://{server}/html/it/main/
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7
    Cookie: ss-opt=temp; X-UAId=; ss-id=sb0aBSSCZRSvraEFjJ6X; ss-pid=OYVgA2o3qOZFjbi5ipx7
    Response headers
    Cache-Control: no-cache
    Transfer-Encoding: chunked
    Content-Type: text/event-stream
    Vary: Accept
    Server: Microsoft-IIS/10.0
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Credentials: true
    X-Powered-By: ARR/3.0
    Persistent-Auth: true
    X-Powered-By: ASP.NET
    Event stream empty

  2. First heartbeat
    Request URL: http://{server}/event-heartbeat?id=AKPtXBz4FlF79VwG26iM
    HTTP method: POST
    HTTP status: 404 / Subscription ‘AKPtXBz4FlF79VwG26iM’ does not exist
    Request headers
    Host: {server}
    Connection: keep-alive
    Content-Length: 0
    Pragma: no-cache
    Cache-Control: no-cache
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
    content-type: text/plain
    Accept: */*
    Origin: http://{server}
    Referer: http://{server}/html/it/main/
    Accept-Encoding: gzip, deflate
    Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7
    Cookie: ss-opt=temp; X-UAId=; ss-id=sb0aBSSCZRSvraEFjJ6X; ss-pid=OYVgA2o3qOZFjbi5ipx7
    Response headers
    Cache-Control: no-cache
    Vary: Accept
    Server: Microsoft-IIS/10.0
    X-Powered-By: ServiceStack/5,50 Net45/Windows
    Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
    Access-Control-Allow-Credentials: true
    X-Powered-By: ARR/3.0
    Persistent-Auth: true
    X-Powered-By: ASP.NET
    Content-Length: 0

So you’re saying Scenario 2 is failing:

Lets format it properly so it’s more readable:

Scenario 2

  1. First event stream

Request URL: http://{server}/event-stream?channels=void&t=1625237024341
HTTP method: GET
Request headers

Host: {server}
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: text/event-stream
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
Referer: http://{server}/html/it/main/
Accept-Encoding: gzip, deflate
Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: ss-opt=temp; X-UAId=; ss-id=sb0aBSSCZRSvraEFjJ6X; ss-pid=OYVgA2o3qOZFjbi5ipx7

Response headers

Cache-Control: no-cache
Transfer-Encoding: chunked
Content-Type: text/event-stream
Vary: Accept
Server: Microsoft-IIS/10.0
X-Powered-By: ServiceStack/5,50 Net45/Windows
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
Access-Control-Allow-Credentials: true
X-Powered-By: ARR/3.0
Persistent-Auth: true
X-Powered-By: ASP.NET

Event stream

id: 1
data: cmd.onConnect {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625237024260","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","id":"AKPtXBz4FlF79VwG26iM","unRegisterUrl":"http://{server}/event-unregister?id=AKPtXBz4FlF79VwG26iM","heartbeatUrl":"http://{server}/event-heartbeat?id=AKPtXBz4FlF79VwG26iM","updateSubscriberUrl":"http://{server}/event-subscribers","heartbeatIntervalMs":"30000","idleTimeoutMs":"60000"}
id: 2
data: void@cmd.onJoin {"userId":"{user id}","isAuthenticated":"true","displayName":"{user name}","channels":"void","createdAt":"1625237024260","profileUrl":"https://raw.githubusercontent.com/ServiceStack/Assets/master/img/apps/no-profile64.png","channel":"void"}
  1. Second event stream

Request URL: http://{server}/event-stream?channels=void&t=1625237024341
HTTP method: GET
Request headers

Host: {server}
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: text/event-stream
Last-Event-ID: 2
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
Referer: http://{server}/html/it/main/
Accept-Encoding: gzip, deflate
Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: ss-opt=temp; X-UAId=; ss-id=sb0aBSSCZRSvraEFjJ6X; ss-pid=OYVgA2o3qOZFjbi5ipx7

Response headers

Cache-Control: no-cache
Transfer-Encoding: chunked
Content-Type: text/event-stream
Vary: Accept
Server: Microsoft-IIS/10.0
X-Powered-By: ServiceStack/5,50 Net45/Windows
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Accept, Content-Type, Authorization, Set-Cookie
Access-Control-Allow-Credentials: true
X-Powered-By: ARR/3.0
Persistent-Auth: true
X-Powered-By: ASP.NET

Event stream empty
3. First heartbeat
Request URL: http://{server}/event-heartbeat?id=AKPtXBz4FlF79VwG26iM
HTTP method: POST
HTTP status: 404 / Subscription ‘AKPtXBz4FlF79VwG26iM’ does not exist

This shows the first event stream connects fine and returns the expected cmd.onConnect and cmd.onJoin responses. But then a second event stream tries to connect, if this is the same client then it means that the first SSE connection request has failed as by the time it sends the first heartbeat to the SSE subscription id of the first request it no longer exists.

Connections can terminate on the server if the Write to the Response fails or from a failed heartbeat which has elapsed. Can you log any errors to see if the SSE connection fails when trying to write to it, e.g:

Plugins.Add(new ServerEventsFeature {
    OnError = (sub,ex) => 
      $"ERROR #{sub.SubscriptionId}, ss-id:{sub.SessionId} {ex.Message}".Print()
});

Hello.
Our application hooks OnConnect, OnSubscribe, OnUnsubscribe, OnError and OnPublish logging all available contextual informations into files.
Watching into these files I did not find logs coming from OnError callback.
Here is a part of the messages written into the just mentioned callbacks.

2021-07-05 16:15:39.547 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [37] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "dkQvr2VUEdUE0YzjEG0T", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:15:39, Channels: "\"void\"" } 
2021-07-05 16:15:39.581 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [37] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "dkQvr2VUEdUE0YzjEG0T", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:15:39, Channels: "\"void\"" } 
2021-07-05 16:18:40.210 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [51] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "ttfZx54reGOBpdmjc4Fd", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:18:40, Channels: "\"void\"" } 
2021-07-05 16:18:40.229 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [51] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "ttfZx54reGOBpdmjc4Fd", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:18:40, Channels: "\"void\"" } 
2021-07-05 16:18:40.250 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [10] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "dkQvr2VUEdUE0YzjEG0T", IsAuthenticated: True, IsClosed: False, LastMessageId: 2, LastPulseAt: 07/05/2021 14:15:39, Channels: "\"void\"" } 
2021-07-05 16:18:43.306 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "tiQglf7UtJEmVeDm8WpT", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:18:43, Channels: "\"void\"" } 
2021-07-05 16:18:43.317 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "tiQglf7UtJEmVeDm8WpT", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:18:43, Channels: "\"void\"" } 
2021-07-05 16:19:10.537 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [64] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "w5LRbfRATkGlQ8H0w9KS", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:19:10, Channels: "\"void\"" } 
2021-07-05 16:19:10.548 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [64] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "w5LRbfRATkGlQ8H0w9KS", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:19:10, Channels: "\"void\"" } 
2021-07-05 16:19:40.327 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [63] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "ttfZx54reGOBpdmjc4Fd", IsAuthenticated: True, IsClosed: False, LastMessageId: 5, LastPulseAt: 07/05/2021 14:18:40, Channels: "\"void\"" } 
2021-07-05 16:19:40.428 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [60] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yLRVXJTeTkeZKn9TZZ55", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:19:40, Channels: "\"void\"" } 
2021-07-05 16:19:40.441 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [60] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yLRVXJTeTkeZKn9TZZ55", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:19:40, Channels: "\"void\"" } 
2021-07-05 16:19:43.432 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [59] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "gFBygEgqEXwrMKfG2Nxz", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:19:43, Channels: "\"void\"" } 
2021-07-05 16:19:43.444 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [59] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "gFBygEgqEXwrMKfG2Nxz", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:19:43, Channels: "\"void\"" } 
2021-07-05 16:19:43.452 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [10] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "tiQglf7UtJEmVeDm8WpT", IsAuthenticated: True, IsClosed: False, LastMessageId: 5, LastPulseAt: 07/05/2021 14:18:43, Channels: "\"void\"" } 
2021-07-05 16:20:01.113 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [76] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "lGqiCc9oZEDI35EREl5k", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:20:01, Channels: "\"void\"" } 
2021-07-05 16:20:01.121 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [76] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "lGqiCc9oZEDI35EREl5k", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:20:01, Channels: "\"void\"" } 
2021-07-05 16:20:10.401 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "CMowtnQQMTDE1bN06jUj", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:20:10, Channels: "\"void\"" } 
2021-07-05 16:20:10.418 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "CMowtnQQMTDE1bN06jUj", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:20:10, Channels: "\"void\"" } 
2021-07-05 16:20:40.323 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [66] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "w5LRbfRATkGlQ8H0w9KS", IsAuthenticated: True, IsClosed: False, LastMessageId: 8, LastPulseAt: 07/05/2021 14:19:10, Channels: "\"void\"" } 
2021-07-05 16:20:40.380 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [67] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "1eNOuMsv7HBh21Fnd4BK", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:20:40, Channels: "\"void\"" } 
2021-07-05 16:20:40.394 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [67] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "1eNOuMsv7HBh21Fnd4BK", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:20:40, Channels: "\"void\"" } 
2021-07-05 16:21:10.274 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [74] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yLRVXJTeTkeZKn9TZZ55", IsAuthenticated: True, IsClosed: False, LastMessageId: 8, LastPulseAt: 07/05/2021 14:19:40, Channels: "\"void\"" } 
2021-07-05 16:21:10.292 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [74] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "lGqiCc9oZEDI35EREl5k", IsAuthenticated: True, IsClosed: False, LastMessageId: 5, LastPulseAt: 07/05/2021 14:20:01, Channels: "\"void\"" } 
2021-07-05 16:21:10.292 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [10] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "gFBygEgqEXwrMKfG2Nxz", IsAuthenticated: True, IsClosed: False, LastMessageId: 7, LastPulseAt: 07/05/2021 14:19:43, Channels: "\"void\"" } 
2021-07-05 16:21:10.398 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [66] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yXzaO0zw46kNp3eMQUTM", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:21:10, Channels: "\"void\"" } 
2021-07-05 16:21:10.414 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [66] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yXzaO0zw46kNp3eMQUTM", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:21:10, Channels: "\"void\"" } 
2021-07-05 16:21:10.429 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [10] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "CMowtnQQMTDE1bN06jUj", IsAuthenticated: True, IsClosed: False, LastMessageId: 7, LastPulseAt: 07/05/2021 14:20:10, Channels: "\"void\"" } 
2021-07-05 16:21:13.406 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [72] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "oftCMSjfhHHzgU4VSAEI", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:21:13, Channels: "\"void\"" } 
2021-07-05 16:21:13.423 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [72] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "oftCMSjfhHHzgU4VSAEI", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:21:13, Channels: "\"void\"" } 
2021-07-05 16:21:40.315 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [67] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "KKRZUpN0P8a44zhrmLGW", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:21:40, Channels: "\"void\"" } 
2021-07-05 16:21:40.331 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [67] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "KKRZUpN0P8a44zhrmLGW", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:21:40, Channels: "\"void\"" } 
2021-07-05 16:21:41.404 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [70] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "7uvZCctvBpB6YIlXk19e", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:21:41, Channels: "\"void\"" } 
2021-07-05 16:21:41.420 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [70] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "7uvZCctvBpB6YIlXk19e", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:21:41, Channels: "\"void\"" } 
2021-07-05 16:21:41.435 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [10] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "1eNOuMsv7HBh21Fnd4BK", IsAuthenticated: True, IsClosed: False, LastMessageId: 9, LastPulseAt: 07/05/2021 14:20:40, Channels: "\"void\"" } 
2021-07-05 16:22:10.464 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [71] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "yXzaO0zw46kNp3eMQUTM", IsAuthenticated: True, IsClosed: False, LastMessageId: 7, LastPulseAt: 07/05/2021 14:21:10, Channels: "\"void\"" } 
2021-07-05 16:22:10.524 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "qMacrQc2GdZMZ5Wuwfi0", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:22:10, Channels: "\"void\"" } 
2021-07-05 16:22:10.545 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [55] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "qMacrQc2GdZMZ5Wuwfi0", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:22:10, Channels: "\"void\"" } 
2021-07-05 16:22:11.447 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [76] [] [{source}] - OnConnect { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "iynCRbuFezo3hGIzB0Om", IsAuthenticated: True, IsClosed: False, LastMessageId: 0, LastPulseAt: 07/05/2021 14:22:11, Channels: "\"void\"" } 
2021-07-05 16:22:11.462 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [76] [] [{source}] - OnSubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "iynCRbuFezo3hGIzB0Om", IsAuthenticated: True, IsClosed: False, LastMessageId: 1, LastPulseAt: 07/05/2021 14:22:11, Channels: "\"void\"" } 
2021-07-05 16:22:40.352 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [75] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "oftCMSjfhHHzgU4VSAEI", IsAuthenticated: True, IsClosed: False, LastMessageId: 8, LastPulseAt: 07/05/2021 14:21:13, Channels: "\"void\"" } 
2021-07-05 16:22:40.370 +02:00 [Infor] [00000000000000000000000000000000] [00000000000000000000000000000000] [75] [] [{source}] - OnUnsubscribe { DisplayName: "{userName}", SessionId: "jRAP8Jkl5t9UDtS52i7a", SubscriptionId: "KKRZUpN0P8a44zhrmLGW", IsAuthenticated: True, IsClosed: False, LastMessageId: 7, LastPulseAt: 07/05/2021 14:21:40, Channels: "\"void\"" }

Are you seeing successful heartbeat responses before clients reconnect to the /event-stream?

If you update to the latest ServiceStack v5.11.1 on MyGet it maintains some additional counters to provide more insight into what’s happening, which you can view a snapshot of in its ServerEvents.GetStats() API.