Troubleshooting RedisResponseException

ServiceStack.Redis
1

I’m using CoreCLR version of ServiceStack.Redis. It’s talking to a single Redis node.

# redis-server --version
Redis server v=3.0.7 sha=00000000:0 malloc=libc bits=64 build=59681bb049f88554

We’re seeing some Redis errors in our logs - for example. They are pretty uncommon (like 0.01% or less), but still 10 seeing about 10 a day.

Unknown reply on multi-request: 42*0
ServiceStack.Redis.RedisNativeClient in ReadMultiData at line 189:0
Byte[][] ReadMultiData()
ServiceStack.Redis.RedisNativeClient in SendReceive at line 206:0
ServiceStack.Redis.RedisNativeClient in SendExpectMultiData
ServiceStack.Redis.RedisClient in GetAllEntriesFromHash
Services.Auth.ClientSpecificCacheService in Hash at line 46:21
Services.Auth.SecurityService in FindRevokedPermissionsForItem at line 145:13
Services.Customers.PatientCaseSearchService in Post at line 42:13
ServiceStack.Host.ServiceRunner`1 in Execute at line 239:0

And another…

Unknown reply on integer response: 79K
ServiceStack.Redis.RedisNativeClient in ReadLong at line 115:0
Int64 ReadLong()
ServiceStack.Redis.RedisNativeClient in SendReceive at line 206:0
ServiceStack.Redis.RedisClient in ContainsKey
Services.Auth.ClientSpecificCacheService in Hash at line 34:17
Services.Auth.SecurityService in FindLocationsAllowingRole at line 115:13
Services.Appointments.AppointmentService in CheckForAppointmentErrors at line 718:17
Services.Appointments.AppointmentService in CheckAppointment at line 676:13
Services.Appointments.AppointmentService in Post at line 163:13
ServiceStack.Host.ServiceRunner`1 in Execute at line 239:0

The Hash code, which accesses Redis is included for reference.

 public Dictionary<int, T> Hash<T>(string key, List<int> ids = null, Func<List<T>> fallback = null, int expiryMins = 10) where T : IHasId
        {
            using (var redis = RedisManager.GetClient())
            {
                if (!redis.ContainsKey(prefix(key)))
                {
                    if (fallback == null)
                        return new Dictionary<int, T>();
                    var result = fallback();
                    redis.SetRangeInHash(prefix(key), result.Select(
                            x => new KeyValuePair<string, string>(x.Id.ToString(), JsonSerializer.SerializeToString(x))).ToArray());
                    redis.ExpireEntryIn(prefix(key), new TimeSpan(0, expiryMins, 0));
                }

                if (ids == null)
                {
                    var hash = redis.GetAllEntriesFromHash(prefix(key));
                    return hash.ToDictionary(x => Convert.ToInt32(x.Key), x => JsonSerializer.DeserializeFromString<T>((string)x.Value ?? "{}"));
                }
                else
                {
                    var keys = new HashSet<string>(ids.Select(x => x.ToString()));
                    var hash = redis.GetAllEntriesFromHash(prefix(key));
                    return hash.Where(x => keys.Contains(x.Key)).ToDictionary(x => Convert.ToInt32(x.Key), x => JsonSerializer.DeserializeFromString<T>((string)x.Value ?? "{}"));
                }
            }
        }

My understanding from some googling that this relates to a lack of thread safety on the BasicRedisClientManager, but I thought the GetClient itself was thread-safe?

Is there anything I’m doing wrong in the Hash code that would cause this, or anything you can advise to diagnose further?

2

Mostly I saw similar exceptions when someone tried to create redis manager each time and then call GetClient() on it. Or when same RedisClient instance was shared between threads and code tries to call redis simultaneously on the same redis client. You can try to look for non-disposed redis clients or clients shared between threads or different managers which are used simultaneously. Also you can try PooledRedisClientManager.
If this issue is the some kind of thread safety issue in RedisClient implementation it should be possible to create stand-alone solution which loads redis with alot of operations and calls GetAllEntriesFromHash in multiple threads and reproduces the exception. It would be good if you can provide some sample which shows the issue.

3

I’m registering a single instance of my redis manger in the container using container.Register(basicRedisClientsManger);

I also use it for the ICacheClient like container.Register<ICacheClient>(c => c.Resolve<IRedisClientsManager>().GetCacheClient());.

Myr exceptions seem to be occurring all over the place in the client, which makes me think it’s something fundamentally up with our usage.

image.png1810×707 162 KB

I’ve tried to reproduce, and failed… will keep going through all my usages of the RedisManager.