Hi there,
I’m using ServiceStack 5.10.4,
and recently update other nuget packages,
“Microsoft.aspnetcore."
"Microsoft.extensions.”
Those were version 3.x, and now 5.0.0.
After make it version 5, ServiceStack didn’t work.
(Exception happen on DefaultMemory.cs,
stream.WriteAsync(bytes, 0, value.Length, token).ConfigAwait()
)
Anyone had this issue? any advice?
What is the full Exception StackTrace? and exactly what packages are you referring to?
It will help to identify the issue if you can post a link to a small stand-alone repro of the issue on GitHub.
@mythz Thanks for the comment!
Full stack trace is like below,
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.CreateResponseHeader(bool appCompleted) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.InitializeResponseAsync(int firstWriteByteCount) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(System.ReadOnlyMemory data, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(byte buffer, int offset, int count, System.Threading.CancellationToken cancellationToken) Unknown
ServiceStack.Text.dll!ServiceStack.Text.DefaultMemory.WriteAsync(System.IO.Stream stream, System.ReadOnlyMemory value, System.Threading.CancellationToken token) Line 500 C#
ServiceStack.dll!ServiceStack.HttpResponseExtensionsInternal.WriteBytesToResponse(ServiceStack.Web.IResponse res, byte responseBytes, string contentType, System.Threading.CancellationToken token) Line 432 C#
ServiceStack.dll!ServiceStack.CompressResponseAttribute.ExecuteAsync(ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse res, object response) Line 78 C#
ServiceStack.dll!ServiceStack.ServiceStackHost.ApplyResponseFiltersSingleAsync(ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse res, object response) Line 326 C#
ServiceStack.dll!ServiceStack.ServiceStackHost.ApplyResponseFiltersAsync(ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse res, object response) Line 241 C#
ServiceStack.dll!ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponseNext(ServiceStack.Web.IRequest httpReq, ServiceStack.Web.IResponse httpRes, object response) Line 116 C#
ServiceStack.dll!ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponse(ServiceStack.Web.IRequest httpReq, ServiceStack.Web.IResponse httpRes, object response) Line 102 C#
ServiceStack.dll!ServiceStack.Host.RestHandler.ProcessRequestAsync(ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse httpRes, string operationName) Line 99 C#
ServiceStack.Kestrel.dll!ServiceStack.AppSelfHostBase.ProcessRequest(Microsoft.AspNetCore.Http.HttpContext context, System.Func<System.Threading.Tasks.Task> next) Line 166 C#
Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> application) Unknown
[Resuming Async Method]
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
[Native to Managed Transition]
[Async Call Stack]
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> application) Unknown
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> httpApplication) Unknown
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.ConnectionDispatcher.Execute(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection connection) Unknown
And not sure which nuget packages specifically, cuz it’s tangled.
I updated nuget packages below,
Microsoft.Extensions.Configuration
Microsoft.Extensions.Configuration.Abstractions
Microsoft.Extensions.Configuration.Binder
Microsoft.Extensions.Configuration.EnvironmentVariables
Microsoft.Extensions.Configuration.FileExtensions
Microsoft.Extensions.Configuration.Json
Microsoft.Extensions.DependencyInjection
Microsoft.Extensions.DependencyInjection.Abstractions
Microsoft.Extensions.FileProviders.Abstractions
Microsoft.Extensions.FileProviders.Physical
Microsoft.Extensions.FileSystemGlobbing
Microsoft.Extensions.Hosting.Abstractions
Microsoft.Extensions.Logging
Microsoft.Extensions.Logging.Abstractions
Anything weird packages here?
I can’t see what the actual Exception Type and Message is?
Does the error still occur if you remove the [CacheResponse] attribute?
@mythz
System.NullReferenceException: ‘Object reference not set to an instance of an object.’
This exception was originally thrown at this call stack:
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.CreateResponseHeader(bool)
And I’m not using CacheResponse attribute.
Sorry I mean [CompressResponse] attribute:
ServiceStack.dll!ServiceStack.CompressResponseAttribute.ExecuteAsync(
ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse res, object response) Line 78
It’s an internal Exception inside Kestrel during writing bytes to the response stream so I’m not going to be able to identify what’s the cause of the Exception without a repro to test against, but what’s the StackTrace without the [CompressResponse] Attribute now?
@mythz
It has same error. below is the stack trace.
I agree with you.
I’m assuming servicestack using Microsoft.AspNetCore.* v2.2, and had some issue with Microsoft.Extensions.* v5.0 ? honestly no idea still.
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.CreateResponseHeader(bool appCompleted) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.InitializeResponseAsync(int firstWriteByteCount) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(System.ReadOnlyMemory data, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(byte buffer, int offset, int count, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(byte buffer, int offset, int count) Unknown
mscorlib.dll!System.IO.StreamWriter.Flush(bool flushStream, bool flushEncoder) Unknown
ServiceStack.Text.dll!ServiceStack.Text.JsonSerializer.SerializeToStream(object value, System.IO.Stream stream) Line 178 C#
ServiceStack.Client.dll!ServiceStack.Serialization.JsonDataContractSerializer.SerializeToStream(object obj, System.IO.Stream stream) Line 70 C#
ServiceStack.dll!ServiceStack.Host.ContentTypes.serializeSync(ServiceStack.Web.StreamSerializerDelegate serializer, ServiceStack.Web.IRequest httpReq, object dto, System.IO.Stream stream) Line 269 C#
ServiceStack.dll!ServiceStack.HttpResponseExtensionsInternal.WriteToResponse(ServiceStack.Web.IResponse response, object result, ServiceStack.Web.StreamSerializerDelegateAsync defaultAction, ServiceStack.Web.IRequest request, byte bodyPrefix, byte bodySuffix, System.Threading.CancellationToken token) Line 350 C#
ServiceStack.dll!ServiceStack.HttpResponseExtensionsInternal.WriteToResponse(ServiceStack.Web.IResponse httpRes, ServiceStack.Web.IRequest httpReq, object result, byte bodyPrefix, byte bodySuffix, System.Threading.CancellationToken token) Line 136 C#
ServiceStack.dll!ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponseNext(ServiceStack.Web.IRequest httpReq, ServiceStack.Web.IResponse httpRes, object response) Line 135 C#
ServiceStack.dll!ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponse(ServiceStack.Web.IRequest httpReq, ServiceStack.Web.IResponse httpRes, object response) Line 102 C#
ServiceStack.dll!ServiceStack.Host.RestHandler.ProcessRequestAsync(ServiceStack.Web.IRequest req, ServiceStack.Web.IResponse httpRes, string operationName) Line 99 C#
ServiceStack.Kestrel.dll!ServiceStack.AppSelfHostBase.ProcessRequest(Microsoft.AspNetCore.Http.HttpContext context, System.Func<System.Threading.Tasks.Task> next) Line 166 C#
Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context) Unknown
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> application) Unknown
[Resuming Async Method]
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
[Native to Managed Transition]
[Async Call Stack]
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> application) Unknown
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context> httpApplication) Unknown
[Async] Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.ConnectionDispatcher.Execute(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection connection) Unknown
It’s a different Exception but still occurring when writing to the response stream. But I wont be able to identify it from here, if you can put a small standalone repro on GitHub I can investigate further.
@mythz
It’s large project and not sure how to repro on small set, but will try. if you have any hints like the attribute, please add here. Thank you very much! and If i create test version, will tag you 
Given that it’s an internal Kestrel exception that occurs during writing to the response stream then only thing I can suggest trying is marking the request to allow sync I/O by adding this to your AppHost:
PreRequestFilters.Add((req,res) => req.AllowSyncIO());
Is there a reason use AllowSyncIO on req not res?
…and still had same error
It’s just used to mark to allow Sync I/O for that request, this should already be added for times when its needed. But it was the only thing I can think of trying in-case you went through a code path where it wasn’t marked.
@mythz
Additional information:
We are running an XUnit unit test within a Framework 4.8 project that is referencing other Framework 4.8 and/or .Net Standard 2.0 class libraries.
They all are using ServiceStack..Core module v 5.10.4. It worked fine until we upgraded Microsoft.AspNetCore. Nuget packages to v5.0.4 and also Microsoft.Extensions.* NuGet packages to v5.0.0.
The failing unit test creates a JsonServiceClient and uses it to make a GET request passing in a request object. It returns the WebException error “The underlying connection was closed: The connection was closed unexpectedly.”
If I put a breakpoint inside the service method, I discover that, even though the client calls Get(…) once, the service method is entered twice. The service method runs fine both times and returns the same results each time, but the client never receives it. AFAIK this is not an authentication challenge issue as there is no authentication configured or set.
I added a GlobalRequestFilter to examine the request object received by the service method each time and they are both identical each time except for the ID. I did the same with a GlobalResponseFilter and likewise they are identical.
Since it shouldn’t happen, I’d investigate why the Service is being called twice. Is the client making 2 requests? You can configure Fiddler to view the HTTP Requests that are sent by the client.
Otherwise see if the call stacks can help identify why it’s being called twice.
Hi @mattk, if you can post it to GitHub and share the link here. Standalone app shouldn’t have anything sensitive in it as it should be just a minimal reproduction of the issue.
Here is a standalone application that replicates the problem.
If you run this code in Visual Studio, it will throw an error when it calls var response = client.Get(new TestRequest());. If you put a breakpoint in TestService.Get() you will see that it gets called twice. Note that this same code worked fine until we upgraded the NuGet packages below (the image shows the NuGet packages BEFORE we updated them).
Thanks we managed to repro the issue with your combination of packages references that resulted in a null reference deep in Kestrel’s internals:
Object reference not set to an instance of an object.","stackTrace":"System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.CreateResponseHeader(Boolean appCompleted)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceStart(Boolean appCompleted)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.InitializeResponseAsync(Int32 firstWriteByteCount)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(ReadOnlyMemory`1 data, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int3 count)
The issue is resolved by using the latest ASP .NET Core 2.x dependencies. Trying upgrade the latest 2.x major versions using the legacy NuGet package references is too slow & cumbersome so I converted your project into a PackageReference project:
Both VS .NET and JetBrains Rider support this with a 1-click right context menu:
Then replacing all dependencies with the default project project references that uses wildcard versions to install the latest 2.x version of each package:
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.Diagnostics" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.Routing" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.*" />
<PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="2.*" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="2.*" />
<PackageReference Include="Microsoft.VisualStudio.Web.BrowserLink" Version="2.*" />
<PackageReference Include="ServiceStack.Core" Version="5.*" />
<PackageReference Include="ServiceStack.Kestrel.Core" Version="5.*" />
</ItemGroup>
Then after deleting App.config & restarting the test project works as expected.
Note: please delete your test project as soon as you read this as it contains your confidential license key.