SerializationException due to network problems?

[h_lenuweit]

Dear ServiceStack support,

being a long (looong) time user of ServiceStack I must say the most positive thing about a library: it does it’s job and won’t get into the way. So thank you.

However recently I get errors from some customers that I couldn’t get a hold of. Some requests coming locally from the server host itself fail, The failing local requests are made using ServiceStack’s JsonServiceClient, while requests coming in through an nginx reverse proxy function as normal. All is (still) running on .NET 4.8, and ServiceStack is on version 5.11.0 for the time being.

The actual error is this (I’m cutting and translating from the stack trace):

2024-01-24 09:30:14,177 | ERROR| docuvita.Server | UnCaughtError: Operation: SetObjectProperties Request: ::1|ServiceStack .NET Client 5.110|http://localhost:36284/server/services/json/reply/SetObjectProperties --Error: SerializationException: Could not deserialize 'application/json' request using docuvita.Contract.Operations.ObjectFunctions.SetObjectProperties'
System.Runtime.Serialization.SerializationException: Could not deserialize 'application/json' request using docuvita.Contract.Operations.ObjectFunctions.SetObjectProperties'
ERROR: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request.

at System.Net.HttpRequestStream.EndRead(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.StreamExtensions.<CopyToNewMemoryStreamAsync>d__50.MoveNext() in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\StreamExtensions.cs:line 675
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Text.DefaultMemory.<DeserializeAsync>d__38.MoveNext() in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\DefaultMemory.cs:line 538
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.ServiceStackHandlerBase.<CreateContentTypeRequestAsync>d__17.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\ServiceStackHandlerBase.cs:line 222
--- End of inner exception stack trace ---
at ServiceStack.Host.Handlers.ServiceStackHandlerBase.<CreateContentTypeRequestAsync>d__17.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\ServiceStackHandlerBase.cs:line 230
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.GenericHandler.<CreateRequestAsync>d__10.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\GenericHandler.cs:line 33
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.GenericHandler.<ProcessRequestAsync>d__12.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\GenericHandler.cs:line 54

Now what’s interesting about this error is that it only occurs on a handful of customer servers. And we could track it down to servers that have an Acronis (Cyber Protect) software product installed. Specifically when you run Acronis’ backup procedure, it leads to above error starting to occur. After a machine reboot, it runs fine until the first backup is performed.

Of course the customers aren’t happy about my suggestion to remove or even only deactivate Acronis. With my limited knowledge about networking and ports, I can see that after a backup, a lot of local ports are being (ab-)used, so it might just be a port exhaustion situation.

Do you have any suggestions on how to proceed? I have found some other articles with similar or the same errors, but with unclear answers:

System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request

In .NET Framework AppSelfHostBase.cs is just a wrapper over AppHostHttpListenerPoolBase.cs, it differs with AppHostHttpListenerBase where instead of executing the Request in the I/O callback thread it instead executes it in a new Thread. One change you could try is have your self-host inherit from AppHostHttpListenerSmartPoolBase from the ServiceStack.NetFramework NuGet package which is a bit more resource efficient as it reuses threads from a more efficient ThreadPool (in case the issue is du…

[mythz]

I can’t see how it can be a port exhaustion issue since the server is already running and listening on a configured TCP port which no other process will be able to listen to. It sounds like the network is being saturated. Don’t really have any other suggestions other than have clients implement an auto retry when requests fail.