C# client heartbeat timeout issue

We have a SSE C# client which using ServiceStack 4.5.6. Sometimes the heartbeats times out which triggers reconnection. We figured out that sometimes the task which sends the heartbeat takes more than 1 minute to finish. (Between log message “Sending Heartbeat…” and “Heartbeat sent to:”) This is causing the server side IdleTimeOut because the client won’t start a new heartbeat unless the previous task finishes and it unsubscribes the user. Then the client gets 404 exception and reconnects to the server. Does anyone have similar issue before?

If you’re changing the Heartbeat Interval you also need to change the IdleTimeout which I recommend to be 2-3x of the HeartbeartIntetval, e.g:

Plugins.Add(new ServerEventsFeature {
    HeartbeatInterval = TimeSpan.FromSeconds(60),
    IdleTimeout = TimeSpan.FromSeconds(180),
})

Thank you for you reply.
We had the default 10 seconds for the heartbeatinterval and 60 seconds for the idletimeout. Is it a common to set the idletimout to several minutes? I think the default is 30 seconds.

I don’t understand why the task to send heartbeat doesn’t return for so long time. (I can see other tasks going on)
I expect the it returns either success or error quite quickly.

        ConnectionInfo.HeartbeatUrl.GetStringFromUrlAsync(requestFilter: req => {
                var hold = httpReq;
                if (hold != null)
                    req.CookieContainer = hold.CookieContainer;

                HeartbeatRequestFilter?.Invoke(req);

                if (log.IsDebugEnabled)
                    log.Debug("[SSE-CLIENT] Sending Heartbeat...");
            })
            .Success(t =>
            {
                if (cancel.IsCancellationRequested)
                    return;

                if (log.IsDebugEnabled)
                    log.Debug("[SSE-CLIENT] Heartbeat sent to: " + ConnectionInfo.HeartbeatUrl);

                StartNewHeartbeat();
            })
            .Error(ex =>
            {
                if (cancel.IsCancellationRequested)
                    return;

                if (log.IsDebugEnabled)
                    log.Debug("[SSE-CLIENT] Error from Heartbeat: " + ex.UnwrapIfSingleException().Message);
                OnExceptionReceived(ex);
            });

The default is 3x over HeartbeatInterval so if it misses 3 heartbeats the connection is considered lost.

The only thing I know of that can block other HttpWeRequests is if you’re making other HttpWebRequests that returns a HttpWebRespons or a stream but aren’t disposing the Response as illustrated in the docs.

You should check that all your Request DTOs have IReturn<T> markers so that you’re not calling the deprecated APIs which return HttpWebResponse, if you are you should use the non-deprecated explicit API in a using statement or ensure each Request DTO has IReturn<T> or IReturnVoid marker interfaces so the response is automatically disposed by the client.

Thanks, mythz!

None of our webservice APIs returns HttpWebResponse or Stream. And the C# client only makes Post. Any communication from the server is though the IReceiver. I am changing all the DTOs to use IReturn. (I didn’t know about this, thanks for the tip)

It’s not what your Server returns, rather if your client requests access to a raw response which it can do for any Service.

If you don’t specify a IReturn<T> then you’ll need to specify the return type at the call-site, e.g:

client.Post<Response>(request);

If you forget to do that you’ll be calling one of the deprecated APIs which returns a HttpWebResponse that you’ll need to explicitly dispose.

Yes, we do specify the response data type when post. Does that mean IReturn is not necessary?

It’s not necessary, but you’d need to make sure you’re not calling any APIs without specifying the response type on each call-site otherwise you need to explicitly dispose the returned HttpWebResponse, using the IReturn* marker interfaces on your Request DTOs would save you from forgetting to do this.

There is an issue with the C# client error handling. It restart automatically whenever an error happens. (Usually an exception) Since there can be multiple tasks running, It could mess up the connection when the exception happens in multiple task, it could stop/start for multiple times and it could end up with two new connections. How can we ensure there is only one connection created?

Please submit an issue about this on https://github.com/ServiceStack/Issues

Issue filed. Thanks.
Another question, is there a way to clean up the defunct auth sessions on the web server? I understand the subscription are removed when it is invalid. But the user sessions will be always there until they are expired. I can remove the session manually from the memory but it doesn’t seems to be an elegant solution. I am wondering is there a easy way to trigger a logout process or something similar?

User Session lifetimes aren’t tied to Server Event Subscriptions, the best approach would be to get your client before it exits to make a call to /auth/logout or via a Service Client:

client.ServiceClient.Post(new Authenticate { provider = "logout" });

sorry I might not explain clearly. I want to deal with the situation where the C# client crashed. For the normal scenario, the client logs out and there is no problem.

You have to detect the crash on the client and call logout explicitly, you don’t want to log the user out on the ServerEventsFeature.OnUnsubscribe event as then they wont be able to re-connect as an authenticated user.

Another question. Dose the C# client automatically support TLS 1.2? As far as I know, on .Net Framework 4.5.2, application needs to manually set ServicePointManager.SecurityProtocol to suppot TLS1.2 or you can enable it by enabling SchUseStrongCrypto in the registry.
Thanks.

The C# Client supports whatever .NET Framework supports, if it works with WebRequest.Create(url) it will work with ServiceStack’s C# Service Clients. So you should be able to enable TLS 1.2 with:

ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;

OK, thanks.
Another question, is there a limit for the number of C# clients connecting to the service stack server? We have tried to connect a large number of users to the server and it stops to authenticate users after around 5000 users logged in.

There’s no technical quota restrictions added by ServiceStack, any restrictions imposed are from ASP.NET limits. You can try tweaking some of the request limits in your Web.config.

Hi mythz, is there any efficient way to search MemoryCacheClient? Currently we have to get all the keys for all user sessions by GetKeysByPattern, then GetAll user sessions and then search the target user sessions in the collection returned.

The other question is, is it possible to define our own key pattern for the auth sessions? It is now always be like “urn:iauthsession:”. and we want to add more information to the key so that we can search it more easily.
Thanks,