Advice on ProcessRequest Error

[lwylie]

Hi,

We unexpectedly had a production server get into a bad state and become non-responsive. It was logging requests with their status codes as expected but all responses were 500 errors but there was no other logging/exceptions from our services. There was one notable error message that came from ServiceStack:

HttpAsyncTaskHandler.ProcessRequest() that should never have been called, was just called from: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at ServiceStack.Host.Handlers.HttpAsyncTaskHandler.ProcessRequest(IRequest httpReq, IResponse httpRes, String operationName)
at ServiceStack.HttpResponseExtensionsInternal.HandleCustomErrorHandler(IResponse httpRes, IRequest httpReq, String contentType, Int32 statusCode, Object errorDto)
at ServiceStack.HttpResponseExtensionsInternal.WriteErrorToResponse(IResponse httpRes, IRequest httpReq, String contentType, String operationName, String errorMessage, Exception ex, Int32 statusCode)
at ServiceStack.ServiceStackHost.HandleUncaughtException(IRequest httpReq, IResponse httpRes, String operationName, Exception ex)
at ServiceStack.HostContext.RaiseAndHandleUncaughtException(IRequest httpReq, IResponse httpRes, String operationName, Exception ex)
at ServiceStack.Host.Handlers.HttpAsyncTaskHandler.HandleException(IRequest httpReq, IResponse httpRes, String operationName, Exception ex)
at ServiceStack.Host.RestHandler.<>c__DisplayClass13_0.b__2(Task t)
at ServiceStack.AsyncExtensions.Continue[TOut](Task task, Func`2 next)
at ServiceStack.Host.RestHandler.ProcessRequestAsync(IRequest httpReq, IResponse httpRes, String operationName)
at ServiceStack.Host.Handlers.HttpAsyncTaskHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

We suspect one of our security scans or system patches is the root cause, but were hoping you could give some insight as to what the error message means, and if you’ve seen if before, what sort of environmental change could cause it.

Restarting IIS and re-deploying our website fixed the problem. Any advice/insight is welcome.

Thanks

Liz

[mythz]

The is just a log warning message for us with StackTrace (i.e. it’s not an Exception) on which code paths are still calling the base ProcessRequest() handler, which was still being called from HandleCustomErrorHandler. The underlying error was likely returning one of the default CustomErrorHttpHandlers:

CustomErrorHttpHandlers = new Dictionary<HttpStatusCode, IServiceStackHandler> {
    { HttpStatusCode.Forbidden, new ForbiddenHttpHandler() },
    { HttpStatusCode.NotFound, new NotFoundHttpHandler() },
};

So it’s likely the underlying Error threw a 403 or 404 Error Response which would’ve been returned to the browser/client. This StackTrace doesn’t contain any info on the Exception itself, just a warning message which I’ve since changed HandleErrorResponse to call ProcessRequestAsync in this commit so it won’t log this message again from v5 that’s now available from MyGet.