Redis Sentinel deadlock issue?


I have the following Redis-related configuration in my global.asax:

// RedisServer is essentially: server1,server2,server3
var sentinel = new RedisSentinel(AppSettings.GetList("RedisServer"))
         RedisManagerFactory = (master, slaves) => new RedisManagerPool(master)
container.Register(c => sentinel.Start());

container.Register<IServerEvents>(c => new RedisServerEvents(c.Resolve<IRedisClientsManager>()));

// Third-party rate limiter plugin that leverages Redis
 Plugins.Add(new RateLimitFeature(Container.Resolve<IRedisClientsManager>()));

I’m also using the AspNetWindowsAuthProvider with a PopulateUserRoles function set.

One of the nodes in my Redis Sentinel had a scheduled reboot… when that happened, I got a RedisResponseException:

ServiceStack.Redis.RedisResponseException: Zero length response
   at ServiceStack.Redis.RedisNativeClient.ParseSingleLine(String r)
   at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisClient.GetValue(String key)
   at ServiceStack.Redis.RedisClient.<>c__DisplayClass130_0`1.<Get>b__0(RedisClient r)
   at ServiceStack.Redis.RedisClient.Exec[T](Func`2 action)
   at ServiceStack.Redis.RedisClientManagerCacheClient.Get[T](String key)
   at ServiceStack.ServiceExtensions.GetSession(IRequest httpReq, Boolean reload)
   at ServiceStack.Auth.AspNetWindowsAuthProvider.PreAuthenticate(IRequest req, IResponse res)
   at ServiceStack.ServiceStackHost.ApplyPreAuthenticateFilters(IRequest httpReq, IResponse httpRes)
   at ServiceStack.ServiceExtensions.GetSession(IRequest httpReq, Boolean reload)

After that, the web server looked deadlocked, all subsequent requests were hung until I restarted the app pool.

I know historically Redis issues stem from misconfiguration, am I doing something wrong in the config?

ServiceStack.Redis performs automatic retries, maybe that’s what was happening, how long was it unresponsive for?

About two hours (that’s when I restarted the app pool).

It’s not that then, not sure what the issue is. The Exception is expected when terminating the redis server, but its not showing the cause of the hanging/deadlock.
Won’t be able to tell without seeing which APIs the threads are being blocked on.