System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request

ServiceStack Self Hosting
1

A few of my users are recieving this error after a while:

System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request

I use ServiceStack version 5.10.2

I am not able to reproduce this error but I found out some points:

  • The error only occurs with “AppSelfHostBase” and not with “AppHostHttpListenerBase”.
  • The error occurs mostly after a couple of hours or at the next morning and in addition there is evidence that this problem is related to the backup tool Arcronis. I checked a few installations and all of them uses Arcronis and the error times fits to the “after backup” times.
  • After recieving that error for the first time, the service ends up in a state, where this erros occurs very often.
  • Restarting the selfhosted service solves this problem (for a couple of hours or to next morning).
  • If the service is in this kind of “error state”, the request “/swagger-ui/” is only partial rendered. (Chrome console shows this error: GET https:/localhost:25851/swagger-ui/swagger-ui.js net::ERR_HTTP2_PROTOCOL_ERROR 200)
  • Nothing to see in “httperr.log” (only “Timer_ConnectionIdle” entrys).

Any ideas?

Two call stack examples:

bei System.Net.HttpResponseStream.EndWrite(IAsyncResult asyncResult)
bei System.IO.Stream.<>c.b__53_1(Stream stream, IAsyncResult asyncResult)
bei System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Text.DefaultMemory.d__34.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Metadata.BaseSoapMetadataHandler.d__5.MoveNext()

bei System.Net.HttpRequestStream.EndRead(IAsyncResult asyncResult)
bei System.IO.Stream.<>c.b__43_1(Stream stream, IAsyncResult asyncResult)
bei System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei System.IO.Stream.d__27.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.StreamExtensions.d__50.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Text.DefaultMemory.d__38.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Host.RestHandler.d__16.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Host.RestHandler.d__15.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei ServiceStack.Host.RestHandler.d__14.MoveNext()

2

Hard to tell but a few possible ideas to look into.

What does your monitored data about disk space say around the time of the crashes? The error in Chrome I think has to do with response data not matching the content-size header so the partial render also matches this. Going by your use of httperr.log, how are you logging files to disk, via stdout or using a logger in process? Are the logging mechanisms the same for selfhost vs httplistener?

I’d have a close look at the state of your servers monitoring logs of system info like diskspace, memory, network/disk IO at the time of the errors to see if anything lines up as well as any other possible differences between how you are using selfhost vs httplistener configured AppHosts.

Part from that, I’m not seeing anything jump out so any more info you can add might help but sounds like server environment related given the intermittency of the problem.

Hope that helps.

1 Like
3

In .NET Framework AppSelfHostBase.cs is just a wrapper over AppHostHttpListenerPoolBase.cs, it differs with AppHostHttpListenerBase where instead of executing the Request in the I/O callback thread it instead executes it in a new Thread.

One change you could try is have your self-host inherit from AppHostHttpListenerSmartPoolBase from the ServiceStack.NetFramework NuGet package which is a bit more resource efficient as it reuses threads from a more efficient ThreadPool (in case the issue is due to lack of resources to execute properly).

1 Like
4

Just want to mention, for anyone else who stumbles across this problem, that the exact same thing happened with Replibit. With versions 5.5.0 and 6.9.0. It works fine at first, but as time went on the application would start to encounter more and more errors over time that would go away with a service restart. It wasn’t RAM or disk space issues, but Replibit was eating all the ports and disabling it stopped the issue from occurring.

2 Likes