Strange issue with concurrent requests, when using a python client

#1

I’m having a strange issue. I’m running SS 5.8 on IIS 10, .NET 4.7.2 (I’ve also tested on 4.8, issue happens there as well). On rare occasions a response from Service Stack is mangled, essentially it looks like the contents of a previous request. I initially suspected a caching issue or some sort of thread-unsafe thing I did, but I was able to reproduce with a python client and a fairly minimal solution running on IIS Express.

I’ve published the solution at https://github.com/almostjulian/ConcurrencyRepro, including a small python client and some batch scripts to generate load continuously. On my machine, the python client will error out on some of the requests (which i’ve color coded in red so you can see) with the some of the contents of a previous request. On rare occasions the response will be so mangled that the python client can’t even deflate the response.

#2

Can you confirm it’s not an issue with the python client, does it happen with any other client?

#3

Yes, I can repro with a .NET core 3.1 client (using RestSharp instead of SS libs, to make it a bit more apples to apples with the python version)

https://github.com/almostjulian/SimpleReproClient

It does take multiple clients to trigger the error, (using a varation of the looping batch scripts I have), but it does happen.

#4

I’m able to repro this with your v5.8 classic ASP .NET project, but not with the latest v5.8.1 in either a classic ASP .NET or .NET Core project that I’m trying to debug this with, so might already be resolved.

Can you try upgrading to v5.8.1 to see if it’s still an issue, this is a small concurrency program I’m using to create concurrent connections:

var responses = new List<string>();
var errors = new List<Exception>();
for (int i = 0; i < 20; i++)
{
    ThreadPool.QueueUserWorkItem(_ => {
        var client = new RestClient("https://localhost:44354");
        foreach (var user in usernames) {
            var request = new RestRequest($"users/{user}.json");                
            var response = client.Get(request);
            try
            {
                dynamic obj = JToken.Parse(response.Content);
                lock(responses) responses.Add($"{obj.userName} {obj.firstName}");
            }
            catch (Exception ex)
            {
                lock(errors) errors.Add(ex);
            }
        }
    });
}
Thread.Sleep(30000);
lock(responses) Console.WriteLine($"success: {responses.Count}");
lock(errors) Console.WriteLine($"errors: {errors.Count}");
#5

Yep, confirmed working in 5.8.1… any ideas what’s different?

As an aside, have you even considered releasing versions that are strictly bugfixes (like this issue?). It make me a bit leery that I now how to move my production environment to a pre-release version in order to resolve an issue. Not a showstopper for me now, but something for you to consider…

In any case, thanks for the quick investigation and turnaround…

#6

It’s been a long time since v5.8, I suspect it’s resolved from issues discovered during gRPC integration testing.

No we’ll only support forward-only releases. AFAIC the current latest pre-release on MyGet is feature complete with what will become v5.9, I’m just trying to finish completing the announcement posts on here to get some feedback on some new features before releasing v5.9.

Note: the pre-releases on MyGet are created with the exact same binaries and go through the same CI testing as official packages on NuGet, which are just a snapshot in time release of the same packages. The primary issue with pre-release packages is if different packages are downloaded at different times (i.e. dirty) where you could get some type/method not found issues which is resolved by clearing your existing NuGet packages cache and re-downloading them which will ensure you’ll have the latest version of each package cached.

1 Like
#7

Understood, thanks, knowing about the CI testing does make me feel better :slight_smile: .

1 Like
Global compression disable?