Strange behaviour with serialisation after upgrade to 5.5

Hi, we were stable on 4.5 for a long time and recently went live, upgrading to 5.5 and started noticing random issues. The first problem was null object properties, were being excluded from serialisation randomly, especially lists properties. And our client apps which expect it to include nulls were throwing errors. Just a small percent of overall volume and we only get this issue in production environment.

Its as if the state of ServiceStack.Text.JSConfig on an instance in the web farm is changed randomly. And some small percentage of responses are serialised as if JsConfig.IncludeNullValues gets reset to false.

Then today we started getting a late bound deserialisation issue. In our message queue handling logic were getting this when we call message.GetBody() (its a Redis based MQ).

  Unable to cast object of type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' to type '...EmailSending'

And theres a DLQ for dictionary. “mq:Dictionary`2.dlq” in Redis.

Its still a small percentage of the overall volume through the email sending queue. When I use the redis-client with lrange “mq:Dictionary`2.dlq” n n - I can see that: __type: “” is missing in the Body object. While others have them.

Any ideas as to what is causing these weird serialisation issues?

Is it possible the JSConfig scope is leaking now in 5.5 in some cases?

When our services call some third party services we also use ServiceStack as a client. For example

 public MandrillGateway(string apiKey)
		{
			this.apiKey = apiKey;
           
			Timeout = TimeSpan.FromSeconds(60);
            UserAgent = "servicestack .net mandrill v1";
		}
		
		class ConfigScope : IDisposable
        {
            private readonly JsConfigScope jsConfigScope;

            public ConfigScope()
            {
	            jsConfigScope = JsConfig.With(new Config {
		            DateHandler = DateHandler.UnixTime,
		            PropertyConvention = PropertyConvention.Lenient,
		            IncludeTypeInfo = false,
		            TextCase = TextCase.SnakeCase
	            });
            } ...

and there’s a few others as well.

Is it this change which has caught us out - https://docs.servicestack.net/releases/v5.4#servicestacktext

Our current settings is like

 public static class SerialisationSettings
    {

        public static void Apply()
        {
            ServiceStack.Text.JsConfig<Guid>.SerializeFn = guid => guid.ToString();
            ServiceStack.Text.JsConfig<Guid?>.SerializeFn = guid => guid.HasValue ? guid.ToString() : string.Empty;
            ServiceStack.Text.JsConfig.IncludeNullValues = true;
...

Can you try with the latest version of v5.6.1 ServiceStack on MyGet which resolves an issue with JsConfigScope in async contexts.

Note: you still wont be able to use JsConfigScope around async/await methods in .NET Framework since it needs to retain the scopes in ThreadStatics. You’ll either need to perform the serialization/deserialization before/after any async calls, or only use sync calls with within a custom jsConfigScope.

If it doesn’t resolve the issue, can you please submit a small stand-alone repro I can run locally to see the issue.

Hi Mythz,

It looks like we do need your recent JsConfigScope fix with v5.6.1. Even after applying the correct implementation in the 5.4 release notes, we can reproduce the problem with repeated calls to /openapi for example.

In production we’re currently on ServiceStack v5.5.0 for most except for ServiceStack.OrmLite.MySql which is 5.1.0. Upgrading to v5.6 seems to require upgrading MySql.Data. We’re currently on 6.10.9.

MySql.Data v8 causes issues with Aurora in production for us and we have already rolled that back. There’s a bunch of open bugs with it. (Most due to null reference exceptions on close of a data reader)

We’ll probably need to switch to that open source version (https://mysqlconnector.net/) but that’s going to need a carefully managed upgrade and testing cycle which we can’t commit to at the moment. We’re under pressure to roll back to Service Stack 4.5.

Do you have any suggestions on a way forwards for us?

Thanks,
Aaron

If you need a custom build of ServiceStack.OrmLite.MySql that references MySql.Data 6.x you can either copy the *.cs classes from ServiceStack.OrmLite.MySql into your project (where it will use your project’s existing MySql.Data reference), or you can create a custom build from a local copy of the project with a different MySql.Data version.

For now we have isolated traffic to /openapi via the ALB which was causing the leakage due to the JsConfigScope bug. Whats the ETA on 5.6.1 (including bug fix) being production ready?

Barring any reported issues (or late additions), the current v5.6.1 on MyGet is what will be deployed in the next v5.7. Aiming for a Release early next week (Monday/Tuesday).

Note: the v5.6.1 passes the same test suite as the official NuGet packages, the only difference between the packages published on NuGet is that they’re a snapshot in time release which will have a v5.7 version that is immutable.

2 Likes