SerializationException due to network problems?

[h_lenuweit]

Dear ServiceStack support,

being a long (looong) time user of ServiceStack I must say the most positive thing about a library: it does it’s job and won’t get into the way. So thank you.

However recently I get errors from some customers that I couldn’t get a hold of. Some requests coming locally from the server host itself fail, The failing local requests are made using ServiceStack’s JsonServiceClient, while requests coming in through an nginx reverse proxy function as normal. All is (still) running on .NET 4.8, and ServiceStack is on version 5.11.0 for the time being.

The actual error is this (I’m cutting and translating from the stack trace):

2024-01-24 09:30:14,177 | ERROR| docuvita.Server | UnCaughtError: Operation: SetObjectProperties Request: ::1|ServiceStack .NET Client 5.110|http://localhost:36284/server/services/json/reply/SetObjectProperties --Error: SerializationException: Could not deserialize 'application/json' request using docuvita.Contract.Operations.ObjectFunctions.SetObjectProperties'
System.Runtime.Serialization.SerializationException: Could not deserialize 'application/json' request using docuvita.Contract.Operations.ObjectFunctions.SetObjectProperties'
ERROR: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request.

at System.Net.HttpRequestStream.EndRead(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.StreamExtensions.<CopyToNewMemoryStreamAsync>d__50.MoveNext() in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\StreamExtensions.cs:line 675
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Text.DefaultMemory.<DeserializeAsync>d__38.MoveNext() in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\DefaultMemory.cs:line 538
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.ServiceStackHandlerBase.<CreateContentTypeRequestAsync>d__17.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\ServiceStackHandlerBase.cs:line 222
--- End of inner exception stack trace ---
at ServiceStack.Host.Handlers.ServiceStackHandlerBase.<CreateContentTypeRequestAsync>d__17.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\ServiceStackHandlerBase.cs:line 230
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.GenericHandler.<CreateRequestAsync>d__10.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\GenericHandler.cs:line 33
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceStack.Host.Handlers.GenericHandler.<ProcessRequestAsync>d__12.MoveNext() in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\Handlers\GenericHandler.cs:line 54

Now what’s interesting about this error is that it only occurs on a handful of customer servers. And we could track it down to servers that have an Acronis (Cyber Protect) software product installed. Specifically when you run Acronis’ backup procedure, it leads to above error starting to occur. After a machine reboot, it runs fine until the first backup is performed.

Of course the customers aren’t happy about my suggestion to remove or even only deactivate Acronis. With my limited knowledge about networking and ports, I can see that after a backup, a lot of local ports are being (ab-)used, so it might just be a port exhaustion situation.

Do you have any suggestions on how to proceed? I have found some other articles with similar or the same errors, but with unclear answers:

System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request

In .NET Framework AppSelfHostBase.cs is just a wrapper over AppHostHttpListenerPoolBase.cs, it differs with AppHostHttpListenerBase where instead of executing the Request in the I/O callback thread it instead executes it in a new Thread. One change you could try is have your self-host inherit from AppHostHttpListenerSmartPoolBase from the ServiceStack.NetFramework NuGet package which is a bit more resource efficient as it reuses threads from a more efficient ThreadPool (in case the issue is du…

[mythz]

I can’t see how it can be a port exhaustion issue since the server is already running and listening on a configured TCP port which no other process will be able to listen to. It sounds like the network is being saturated. Don’t really have any other suggestions other than have clients implement an auto retry when requests fail.

[h_lenuweit]

Thanks Demis for your quick reply. Unfortunately we still encounter the problem, so we have conducted lots of tests to track down the issue.

Sending repeated requests until they go through might work but would clutter the log with errors (as the server reports said SerializationException).

Here’s what we tried:

• We start our service, and then perform an Acronis backup
• (By the way this Acronis stuff permanently runs dozens of processes/services and makes connections as hell, also originating locally. So the common ground here is both our service and Acronis stuff are making connections originating and ending locally).
• Approx 2mins after starting the backup, the error starts to show up on our service. From then on, the error occurs regularly, but not on all requests.
• No useful information in Acronis’s own logs or in Windows event log.
• It makes no difference if we run our requests in a locked/critical section, it only slows things down
• Deactivating KeepAlive on JsonServiceClient’s request filter also makes no difference
• We even tried to hook into BindIPEndPointDelegate to give it a specific port range at no avail
• tweaking all sorts of TCP/IP settings in Windows registry doesn’t change the behavior
• Switching between AppSelfHostBase and AppSelfHostBase didn’t make a difference
• We added lots of logging, and what we see is:
  • Client-side dumping of the request JSON reveals healthy data
  • No dumping was possible on server side (content length -1). OK makes sense due to SerializationException

==> So clearly the connection could be made, but then - it’s terminated prematurely? Any way to dig deeper into what’s happening?

Ah one important thing. The requests I’m talking about come from the same process (our service). They are used by plugins which can make API calls. Theoretically we could circumvent the network parts by injecting those requests “somewhere” directly within the process. Could that be done somehow while maintaining the whole request/response pipeline?

[mythz]

I’ve previously suspected that it could be a network saturation issue but I’m not seeing anywhere in your findings where you’ve profiled and ruled it out? If 2 independent OS processes were interfering with each other first thing I’d look at is which server resource utilization is growing higher under load, i.e. Network, CPU and Memory.

Have you tried asking the Acronis support team if they’ve had other customers that have experienced this issue?

@h_lenuweit

Theoretically we could circumvent the network parts by injecting those requests “somewhere” directly within the process. Could that be done somehow while maintaining the whole request/response pipeline?

I don’t really understand what this is asking, do you want to execute the request on a different Thread to the HTTP Request Worker Thread? To improve utilization you should change any blocking I/O to use non-blocking async APIs. You could run logic on a different thread then wait for it to complete before returning the request but I don’t see how that would help as that’s just going to add overhead.

The Exception StackTrace shown is thrown when trying to read from the Request Input Stream, anything you do within the Service implementation isn’t going to help if you’re not able to read the incoming Request Body. The Request Stream is forward only so you wont be able to reread it, so IMO only solution would be to have clients auto retry failed requests.

@h_lenuweit

• Switching between AppSelfHostBase and AppSelfHostBase didn’t make a difference

I’m not sure if this means that you’ve also tried running it on ASP .NET Core, if you did one other thing you could try is telling ASP .NET Core to buffer the request:

app.Use(async (context, next) => {
    context.Request.EnableBuffering();
    await next();
});