JsonHttpClient POST method hangs in v5.10.4

Clients
1

I’ve hit a client glitch using JsonHttpClient.Post<IReturnVoid>() to send a multi-part file upload request.

I have code that worked using ServiceStack.Client v5.9.0 but no longer works using v5.10.4.

The JsonHttpClient.Post<IReturnVoid>() method fails to return once the response is received. It just hangs, and the client process needs to be terminated.

The endpoint responding to the request is very non-standard and could not be used with the stock JsonHttpClient.PostFileWithRequest() method. So I had to write my own FileUploader class to consume this weird API, and I based it on the ServiceStack client code, and then started hacking. It’s been working fine for a few years, on a few of this system’s esoteric file-upload APIs, but I discovered that this one API has caused clients using ServiceStack 5.10.4 to break since it was released in January.

This particular API responds with a 201 (No Content) and a Location header with a URL the client should follow, like a redirect.

The entire purpose of the FileUploader class is to issue the POST multi-part file and use a ResultsFilterResponse to save any received Location header value to a string property.

Here is an example request sent by this code:

POST https://scc.aqsamples.com.au/api/v2/observationimports/dryrun?fileType=SIMPLE_CSV&timeZoneOffset=-07%3a00&linkFieldVisitsForNewObservations=True HTTP/1.1
Authorization: token 01234567890123456789
User-Agent: ServiceStack .NET Client 5.104/Aquarius.Client 20.3.1.0/LabFileImporter 1.0.0.0
Accept: application/json
Content-Type: multipart/form-data; boundary="e87a7ba1-c7d6-41a9-90fe-f00681a354f7"
Host: scc.aqsamples.com.au
Content-Length: 8396
Expect: 100-continue
Accept-Encoding: gzip, deflate

--e87a7ba1-c7d6-41a9-90fe-f00681a354f7
Content-Disposition: form-data; name=file; filename="LabFileImporter (v1.0.0.0) Uploads.csv"
Content-Type: text/csv

Observation ID,Location ID,Observed Property ID,Observed DateTime,Analyzed DateTime,Depth,Depth Unit,Data Classification,Result Value,Result Unit,Result Status,Result Grade,Medium,Activity ID,Activity Name,Collection Method,"Field: Device ID","Field: Device Type","Field: Comment","Lab: Specimen Name","Lab: Analysis Method","Lab: Detection Condition","Lab: Limit Type","Lab: MDL","Lab: MRL","Lab: Quality Flag","Lab: Received DateTime","Lab: Prepared DateTime","Lab: Sample Fraction","Lab: From Laboratory","Lab: Sample ID","Lab: Dilution Factor","Lab: Comment","QC: Type","QC: Source Sample ID"
,05MOC,Temperature,2020-04-22T09:00:00.000-07:00,2020-04-22T09:00:00.000-07:00,,,FIELD_RESULT,24.8,DegC,Preliminary,,Saline Water,,,,,,,,,,,,,,,,,,,,,,
< 20 rows removed >
,04MIC,Chloroa Phenophytin - ratio,2020-04-22T09:20:00.000-07:00,2020-04-22T09:20:00.000-07:00,,,LAB,1,ratio,Requested,,Saline Water,,22Apr2020-04MIC-09:20-REPLICATE,,,,,Properties,CM37;Chlorophyll a;Unity Water,,,,,,,,,15,,,,REPLICATE,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
--e87a7ba1-c7d6-41a9-90fe-f00681a354f7--

And here is the HTTP response, received within a second.

HTTP/1.1 201 Created
Date: Wed, 31 Mar 2021 22:07:23 GMT
Connection: keep-alive
Server: nginx/1.15.9
Cache-Control: no-cache, private, max-age=0, no-store
Location: https://scc.aqsamples.com.au/api/v2/observationimports/393b4bfc-f740-430f-8a39-f6756e01cb3b/status
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=316224000
Content-Security-Policy: default-src https:; script-src 'self' 'unsafe-inline' https://www.google-analytics.com; style-src 'self' 'unsafe-inline'; img-src https: data: blob:
Content-Length: 0

But the JsonHttpClient.Post() method does not return, nor does the ResultsFilterResponse callback get invoked.

When paused within Visual Studio, the stack trace is as follows. Any ideas on what to try next?

mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task<ServiceStack.IReturnVoid>.GetResultCore(bool waitCompletionNotification)
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.Result.get()
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]
ServiceStack.HttpClient!ServiceStack.InternalExtensions.GetSyncResponse<System.__Canon>(System.Threading.Tasks.Task<System.__Canon> task) Line 1169
	at C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack.HttpClient\JsonHttpClient.cs(1169)
ServiceStack.HttpClient!ServiceStack.JsonHttpClient.Post<ServiceStack.IReturnVoid>(string relativeOrAbsoluteUrl, object request) Line 919
	at C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack.HttpClient\JsonHttpClient.cs(919)
Aquarius.Client!Aquarius.Samples.Client.FileUploader.PostFileWithRequest<System.__Canon>(string relativeOrAbsoluteUri, System.IO.Stream contentToUpload, string uploadedFileName, System.__Canon requestDto, System.Net.Http.HttpContent extraContent, string extraContentName)
Aquarius.Client!Aquarius.Samples.Client.SamplesClient.PostFileWithRequest.AnonymousMethod__0()
Aquarius.Client!Aquarius.Samples.Client.SamplesClient.InvokeWebServiceMethod.AnonymousMethod__0()
Aquarius.Client!Aquarius.Samples.Client.SamplesClient.InvokeWebServiceMethod<int>(System.Func<int> webServiceMethod, System.Func<ServiceStack.Text.JsConfigScope> scopeMethod)
Aquarius.Client!Aquarius.Samples.Client.SamplesClient.InvokeWebServiceMethod(System.Action webServiceMethod, System.Func<ServiceStack.Text.JsConfigScope> scopeMethod)
Aquarius.Client!Aquarius.Samples.Client.SamplesClient.PostFileWithRequest<Aquarius.Samples.Client.ServiceModel.PostObservationsDryRunV2>(System.IO.Stream contentToUpload, string uploadedFileName, Aquarius.Samples.Client.ServiceModel.PostObservationsDryRunV2 requestDto, System.Net.Http.HttpContent extraContent, string extraContentName)
LabFileImporter.exe!LabFileImporter.ImportClient.PostImportDryRunForStatusUrl(string filename, byte[] contentBytes) Line 58
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\ImportClient.cs(58)
LabFileImporter.exe!LabFileImporter.Importer.ImportObservationsToSamples(System.Collections.Generic.List<LabFileImporter.ObservationV2> observations) Line 126
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\Importer.cs(126)
LabFileImporter.exe!LabFileImporter.Importer.Import() Line 40
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\Importer.cs(40)
LabFileImporter.exe!LabFileImporter.MainForm.ImportFiles(string[] paths) Line 260
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\MainForm.cs(260)
LabFileImporter.exe!LabFileImporter.MainForm.TryImportFiles(string[] paths) Line 228
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\MainForm.cs(228)
LabFileImporter.exe!LabFileImporter.MainForm.importButton_MouseClick(object sender, System.Windows.Forms.MouseEventArgs e) Line 206
	at C:\git\Examples\Samples\DotNetSdk\LabFileImporter\MainForm.cs(206)
2

The async libraries have had an async overhaul to use async/await and .ConfigureAwait() internally everywhere which has caused some JsonHttpClient “sync over async” requests to hang.

Change all your sync API requests to use JsonServiceClient instead or change them to use Async APIs when using JsonHttpClient.

3

Thanks, I’ll give that a try.

4

Could you elaborate a bit on this?

We recently upgraded from 4.x to 5.11.0 and afterwards consistently hit this issue making a single call (the same call every time). Switching all calls to use JsonServiceClient was not an option at this time, so I temporarily unblocked by changing just that single call to use the JsonServiceClient. However, I’d like to understand what’s going on in order to determine what to do here longer term. Is this a bug in ServiceStack? If so, is there an issue tracking it, any guess as to when it might be fixed, etc? Or is the bug somewhere else (e.g. in Microsoft’s HttpClient implementation or something)? Is there a reason to believe that switching to using the async APIs (and then just calling .Result to wait for it to finish) would be any different than what’s currently happening…?

5

It’s as written, the async APIs have been rewritten to use the recommended async/await pattern with .ConfigureAwait(false) (recommended for general purpose libraries) the side-effect of which caused documented issues with “sync over async” requests, the issue is that you should never do blocking sync over async. The correct solution is to not block on async APIs, i.e. only use async APIs on JsonHttpClient as every sync request on .NET’s HttpClient has to do “sync over async” since it doesn’t provide any way to make true blocking sync calls.