StackOverflow exception on dotnet.exe on .Net Core

Rob · February 14, 2017, 6:21pm

Symptom:

Hosting environment: Production
Content root path: D:\XXXXX
Now listening on: http://localhost:29127
Application started. Press Ctrl+C to shut down.

Process is terminated due to StackOverflowException.

The dotnet.exe runs for some time and then - all of the sudden - crashes. Sometimes it recovers, but sometimes it does not. In the latter case, I need to manually kick the IIS app to get it back.

This StackOverflow exception occurs when a response is written to the stream. The bottom part of the stack strace:

…

ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteListsOfElements`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteGenericIList(System.IO.TextWriter, System.Collections.Generic.IList`1<System.__Canon>)+f9 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteListsOfElements`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteGenericIList(System.IO.TextWriter, System.Collections.Generic.IList`1<System.__Canon>)+f9 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteListsOfElements`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteGenericIList(System.IO.TextWriter, System.Collections.Generic.IList`1<System.__Canon>)+f9 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteListsOfElements`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteGenericIList(System.IO.TextWriter, System.Collections.Generic.IList`1<System.__Canon>)+f9 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d** 
ServiceStack.Text.JsonSerializer.SerializeToStream(System.Object, System.Type, System.IO.Stream) 
ServiceStack.Text.JsonSerializer.SerializeToStream[[System.__Canon, System.Private.CoreLib]](System.__Canon, System.IO.Stream) 
ServiceStack.HttpResponseExtensionsInternal.WriteToResponse(ServiceStack.Web.IResponse, System.Object, ServiceStack.Web.ResponseSerializerDelegate, ServiceStack.Web.IRequest, Byte[], Byte[]) 
ServiceStack.HttpResponseExtensionsInternal.WriteToResponse(ServiceStack.Web.IResponse, ServiceStack.Web.IRequest, System.Object, Byte[], Byte[]) 
ServiceStack.Host.RestHandler+<>c__DisplayClass13_0.<ProcessRequestAsync>b__0(System.Object) 
ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponse(System.Object, System.Func`2<System.Object, System.Threading.Tasks.Task>, System.Func`2<System.Exception, System.Threading.Tasks.Task>) 
ServiceStack.Host.RestHandler.ProcessRequestAsync(ServiceStack.Web.IRequest, ServiceStack.Web.IResponse, System.String) 
ServiceStack.AppHostBase.ProcessRequest(Microsoft.AspNetCore.Http.HttpContext, System.Func`1<System.Threading.Tasks.Task>) 
Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext) 
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware+<Invoke>d__6.MoveNext() 
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware+<Invoke>d__6, Microsoft.AspNetCore.Diagnostics]](<Invoke>d__6ByRef) 
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext) 
Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware+<Invoke>d__8.MoveNext() 
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware+<Invoke>d__8, Microsoft.AspNetCore.Server.IISIntegration]](<Invoke>d__8ByRef) 
Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext) 
Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware+<Invoke>d__3.MoveNext() 
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware+<Invoke>d__3, Microsoft.AspNetCore.Hosting]](<Invoke>d__3ByRef) 
Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext) 
Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Context) 
Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1+<RequestProcessingAsync>d__2[[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context, Microsoft.AspNetCore.Hosting]].MoveNext() 
System_Private_CoreLib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+6e 
System_Private_CoreLib_ni+61a5bb 
Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.LoggingThreadPool.<RunAction>b__5_0(System.Object) 
System_Private_CoreLib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+6e 
System_Private_CoreLib_ni+747084 
System_Private_CoreLib_ni+6176ac 
coreclr!CallDescrWorkerInternal+83 
coreclr!MethodDescCallSite::CallTargetWorker+14e 
coreclr!QueueUserWorkItemManagedCallback+38 
coreclr!SimpleComCallWrapper::InitNew+313 
coreclr!SimpleComCallWrapper::InitNew+268 
coreclr!SimpleComCallWrapper::InitNew+195 
coreclr!SimpleComCallWrapper::InitNew+107 
coreclr!ManagedPerAppDomainTPCount::DispatchWorkItem+8e 
coreclr!ThreadpoolMgr::WorkerThreadStart+188 
coreclr!Thread::intermediateThreadProc+8b 
kernel32!BaseThreadInitThunk+1a 
ntdll!RtlUserThreadStart+1d

The trace goes on with the Common.WriteType and Common.WriteListOfElements’2 for some time, before it quits and calls it a day. I have not been able to get more info than this, and I am basically scratching my head… Any ideas to get this sorted are more than welcome… There seems to be some recursive loop. Is there any possibility to get more info than this?

mythz · February 14, 2017, 6:27pm

I’m assuming this is due to the DTO that’s being serialized. What does your Response DTO that causes this StackTrace look like?

Rob · February 14, 2017, 6:33pm

That’s the thing. I have a large number of DTO’s being serialized. I figured this was the issue, but then again… Shouldn’t the serializer call it a day after 50 iterations (maxdepth)? Any pointers on how I can find the DTO causing this?

Also, if the exception ocuurs: can I trap it without the runtime crashing? With that I can track back to the actual DTO.

Rob · February 14, 2017, 6:50pm

The last bit of the stack at the time of the crash:

.NET Call Stack

DomainBoundILStubClass.IL_STUB_PInvoke(Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvAsyncHandle)+9a 
[[InlinedCallFrame] (Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv+NativeMethods.uv_async_send)] Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv+NativeMethods.uv_async_send(Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvAsyncHandle) 
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.async_send(Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvAsyncHandle)+13 
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.Post(System.Action`1, System.Object)+e6 
Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketOutput.WriteAsync(System.ArraySegment`1, System.Threading.CancellationToken, Boolean, Boolean, Boolean, Boolean)+4d4 
Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketOutput.Microsoft.AspNetCore.Server.Kestrel.Internal.Http.ISocketOutput.Write(System.ArraySegment`1, Boolean)+28 
Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.Write(System.ArraySegment`1)+c3 
Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameResponseStream.Write(Byte[], Int32, Int32)+7e 
System_Private_CoreLib_ni!System.IO.StreamWriter.Flush(Boolean, Boolean)+b9 
System_Private_CoreLib_ni!System.IO.StreamWriter.Write(System.String)+3f 
ServiceStack.Text.Json.JsonTypeSerializer.WriteRawString(System.IO.TextWriter, System.String)+2e 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+508 
ServiceStack.Text.Common.WriteListsOfElements`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteGenericIList(System.IO.TextWriter, System.Collections.Generic.IList`1)+f9 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d 
ServiceStack.Text.Common.WriteType`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].WriteProperties(System.IO.TextWriter, System.Object)+59d

So, there seems to be an end to it on JsonTypeSerializer.WriteRawString.

mythz · February 14, 2017, 7:01pm

It’s not likely you’ll be able to catch a StackOverflowException.

This code-path looks like it bypasses the MaxDepth count, I’d think the best way to find the DTO may be to emit the response dto type in a response filter, e.g:

GlobalResponseFilters.Add((req, res, response) =>
{
    if (response != null)
    {
        LogManager.GetLogger(response.GetType()).Debug("Response: " + response.GetType().Name);
    }
});

Then look at what Type is logged before the StackOverflowException.

Rob · February 14, 2017, 7:20pm

OK, will do and let you know if this is helpful. THX!

mythz · February 14, 2017, 7:21pm

As for the DTO I’d be looking for is any DTO with cyclical dependencies, from the StackTrace it looks like it’s from a DTO with a List<ChildDto> and that ChildDto has either a cyclical reference or another list of ChildDtos with a cyclical reference.

Rob · February 14, 2017, 8:08pm

I tried to get around that kind of DTO, but I may have missed one. Implemented full logging - easy - and I see some weird requests coming in. Like someone is trying to do some serious bad…
Thanks so far. I think I have enough pointers now to get around this.

Rob · February 15, 2017, 6:40am

Replying to my own message.
The logging proved not helpful, so I decided to dig until I found a doubl-referencing DTO. There were several. After a long night I have pushed a new version live with non-cyclic DTOs. No crash so far…

Rob · February 17, 2017, 6:45am

I can confirm that the circular DTO reference caused the issue. The big issue here is that this error in not caught by any error handling mechanism I could find and it crashes the dotnet.exe runtime. As a side effect, the site goes down and (sometimes) does not get back up again. Any chance of trapping this in AppHost with an exception handler?

mythz · February 17, 2017, 8:05am

You can’t trap StackOverflowExceptions that’s why it brings the host down.