Application crashing

ServiceStack Server Events
1

I’ve had an application happily using server sent events for a while, however of late I’ve begun to notice that the application is crashing completely.

The logging in the application doesn’t seem to catch anything, however in the event log in Windows I see the following message:

Application: Application.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidCastException
at ServiceStack.HttpUtils.SendStringToUrlAsync(System.String, System.String, System.String, System.String, System.String, System.Action1<System.Net.HttpWebRequest>, System.Action1<System.Net.HttpWebResponse>)
at ServiceStack.ServerEventsClient.Heartbeat(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

The ServerEventsClient is setup with default options. Let me know if there’s any further information you need.

2

This is going to be extremely hard to diagnose without a repro.

If you haven’t upgraded to the latest version please do and confirm if the Error keeps happening. How often does this error occur? What platform are you running the ServerEventsClient on? e.g. Windows 2012 .NET 4.5 / Ubunto + Mono / iOS / Android, etc.

The Stack Trace suggests the Exception originated from ServerEventsClient, can you provide the full initialization and usage code you’re using for ServerEventsClient as well.

3

Upgraded to the latest version and it’s still occurring. However, I have tracked down what’s causing it.

When I setup the ServerEventsFeature I add an .OnConnect sub, this replaces the heartbeat, unregister and update subscriber URLs and removes the http/https so that the Javascript client can use http/https depending on what protocol the page has been fetched with.

I think this is causing a problem with the .NET ServerEventsClients:

Additional information: Unable to cast object of type ‘System.Net.FileWebRequest’ to type ‘System.Net.HttpWebRequest’.

I think it’s seeing the //domain.com/server-stream as a file request rather than http/https.

4

Yeah that would cause it. ServiceStack should already be using the most appropriate absolute URL scheme depending on what URL the /event-stream was called with. Also in the last v4.5.0 release ServiceStack will automatically use https if it’s behind a SSL-terminating proxy forwarded with a X-Forwarded-Proto=https HTTP Request Header.

Otherwise you can override AppHost.ResolveAbsoluteUrl() and use your own heuristic to replace the url with the preferred scheme.

5

Just to let you know I removed the code I’d added and the v4.5.0 release fixes the issue with https connections.

Thanks so much for your time and for such an awesome product!

1 Like
6

Great! Thanks so much.
Can I ask you another question? Do you know if servicestack C# client support SSL/TLS? Is there any sample code? My understanding is the client needs to send the certificate with the authentication request, correct?
Sorry, I am quite new to servicestack.

7

The JsonServiceClient uses .NET WebRequest which automatically uses SSL if the url starts with https://. If the server uses a trusted certificate it should just work otherwise you may need to implement your own SSL Validation check to control which certificates you want to allow.

The JsonHttpClient uses HttpClient behind the scenes, here’s some info on configuring to use SSL. You can customize either globally:

JsonHttpClient.GlobalHttpMessageHandlerFactory = msgHandler => ...;

Or per instance:

var client = new JsonHttpClient(baseUrl);
client.HttpMessageHandler = msgHandler => ...;
8

Thanks mythz!

I found some issue related to the original question. Basically in the C# client, we registered a UserCommandReceiver to consume the messages from the server. If any of the callbacks in the receiver throws an exception, the SSE client stops receiving anything from the server. (though the general OnException doesn’t catch that exception) Is this an expected behavior? For sure we will fix the exception issue but I thought each callback is a task and even an exception is thrown in the task, it won’t make the service stack client not functioning.

9

Well it’s expected that you handle User Exceptions in your own callbacks, the OnException handler was only meant for handling Exceptions originating internally from the ServerEventsClient itself (which would trigger an auto-reconnect), but it may provide better behavior to also catch and trigger the OnException callback for Unhandled User Exceptions as well, which I’ve just added in this commit.

This change is available from v4.5.7 that’s now available on MyGet.

10

A question about sending the message from C# client to server. Currently we are using the ServiceClient of the ServerEventsClient to Post messages to the server service. Is this the desired way? Will it affect the performance of the SSE?

11

Also, I noticed if the heartbeat happens when there is server is trying to notify some messages to the client, the heartbeat response can be very slow. (almost 1 minute, during that period, nothing happens in the tracing log) Is it possible that some collision happens on the communication channel?

12

Server Events enables 1-way communication from Server to client, you’d still use the Service Clients to call Services on the Server. This doesn’t affect SSE performance which is just a long-lived HTTP connection.

13

Separate .NET Web Requests are used but .NET still pools the connections behind the scenes and has a default limit of 2 connections back to the same server. You can increase the default connection limits in your Web.config:

<?xml version="1.0" encoding="utf-8" ?> 
<configuration> 
  <system.net> 
    <connectionManagement> 
      <add address="*" maxconnection="1000" /> 
    </connectionManagement> 
  </system.net> 
</configuration>

Or configure it in C# with:

ServicePointManager.DefaultConnectionLimit = 1000;

Alternatively you could use the JsonHttpClient which uses a separate connection pool