Parallel requests fail, ServiceStack Redis Error Unexpected reply: *2

1

Sequential requests work fine. But when parallel batch requests are made, it starts giving error for some requests as follow
Unexpected reply: *2

I have created PooledRedisClientManager instance using static variable. I can not register and resolve PooledRedisClientManager in container since I have to make multiple instances of PooledRedisClientManager to connect to different redis servers.

dictClientManagers is static dictionary. This is how I’m initializing manager

RedisClientManagerConfig conf = new RedisClientManagerConfig(); conf.DefaultDb = 0; conf.MaxWritePoolSize = 5; conf.MaxReadPoolSize = 5; string[] masters = new string[] { connStringMaster }; string[] slaves = new string[] { connStringSlave1 }; dictClientManagers[redisServerName] = new PooledRedisClientManager(readWriteHosts: masters, readOnlyHosts: slaves, config: conf);
Any help would be really appreciated as It is blocking us to push the code to the production.

Thanks

2

This error happens when the Redis Client is in an invalid state which is almost always a result of using the same non-ThreadSafe Redis Client instance across multiple threads.

Make sure you’re using a ThreadSafe Redis Client Manager and are always retrieving and disposing of the Redis Client in the same thread, are never sharing the RedisClient instance across multiple threads, not storing it in a static property, etc.

3

Thanks. It seems issue is because of my static managers.

How do you recommend to create managers for different redis servers in same application ?

4

You can hold the Thread Safe client managers in a static variable (I.e. they should only be a single instance of them) then all your different threads can access the client manager to resolve their own Redis client instance which they should dispose after they’ve finished using them to get their instance released back into the pool.

5

I guess I’m implementing it same way. Please have a look on my code.

Please let us know if you feel any changes are required to be made.

public static class Redis
{
    private static PooledRedisClientManager gdsPool = null;
    public static void selectNInitRedis()
    {
        string clientName = getClientName();                 
        string connStringMaster = clientName + ":" + ConfigurationManager.AppSettings["REDIS_GDS_MASTER"].ToString();
        string connStringSlave1 = clientName + ":" + ConfigurationManager.AppSettings["REDIS_GDS_SLAVE1"].ToString();
        RedisClientManagerConfig conf = new RedisClientManagerConfig();
        conf.DefaultDb = 0;
        conf.MaxWritePoolSize = 3;
        conf.MaxReadPoolSize =3;
        string[] masters = new string[] { connStringMaster };
        string[] slaves = new string[] { connStringSlave1 };                
        gdsPool= new PooledRedisClientManager(readWriteHosts: masters, readOnlyHosts: slaves, config: conf);
        Logging.Loggers.RedisLogger.info("created pool " + redisServer.ToString());                
    }
    public static IRedisClient getClient(SlaveOrMaster slaveOrMaster = SlaveOrMaster.SLAVE1)
    {
        switch (slaveOrMaster)
        {                    
            case SlaveOrMaster.MASTER:
                return gdsPool.GetClient();
            case SlaveOrMaster.SLAVE1:
            default:
                return gdsPool.GetReadOnlyClient();
        }            
    }
}

//Initiliazation 
public static class WebApiConfig
{
    public static void Register(HttpConfiguration config)
    {
        //GDS Redis Init
        Domain.Lib.DB.Redis.selectNInitRedis();

        //Routes registration ...

    }
}

//Usage
public Dictionary<string,string> getBusesDict(int fromCityId, int toCityId, DateTime journeyDate)
{
    IEnumerable<KeyValuePair<string, string>> busesIterator = null;
    string[] eVars = new string[] { journeyDate.Date.ToString("yyyy-MM-dd"), fromCityId.ToString(), toCityId.ToString() };
    string redisKey = this.getRedisKey(eVars);
    Dictionary<string, string> dictResult = new Dictionary<string, string>(); ;
    using (var client = Redis.getClient())
    {
        foreach( var item in client.ScanAllHashEntries(redisKey))
        {
            dictResult.Add(item.Key,item.Value);
        }                
    }
    return dictResult;
}
6

The important part is that you’re retrieving and disposing the Redis Client instance which is good. I’d only be changing the defaults if you need to as the Max Pool Size looks really low.

I don’t see value in the additional boilerplate and would just be accessing the Redis Manger directly, e.g:

public static class Redis
{
    public static IRedisClientsManager Manager { get; set; }
}

using (var client = Redis.Manager.GetClient())
{
    foreach (var item in client.ScanAllHashEntries(redisKey))
    {
        dictResult.Add(item.Key,item.Value);
    }                
}

Although my personal preference is to use an IOC to inject the Redis Manager into the classes that need it which will make it more testable.

On a style-note, the .NET Framework design guidelines requires the use of PascalCase for public methods.

7

I have increased MaxReadPoolSize and MaxWritePoolSize to 20 and tested the load with 10 parallel threads.
Still getting same error. 'Unexpected reply: *2'

I have added additional wrapper to access manager because I need different managers for different kind requests. So that I have to manage one code to access different set of redis servers.

Thanks for style-note, I’ll keep that in mind.

8

Please add a stand-alone example that we can run locally to see the issue. (e.g. on GitHub)

Note when using any Scan* APIs that return IEnumerable you shouldn’t be consuming it lazily after the client has been disposed (and return back to the pool) as the IEnumerable would still be calling the Redis client that has been returned to the pool. It’s not an issue in your example, just in-case you’re using the Scan APIs elsewhere and consuming it lazily.

9

Yes. I’m using the same way in another code like following

 public IEnumerable<KeyValuePair<string, string>> GetFares(int fromCityId, int toCityId, DateTime journeyDate)
        {
            IEnumerable<KeyValuePair<string, string>> busesIterator = null;
            string[] eVars = new string[] { journeyDate.Date.ToString("yyyy-MM-dd"), fromCityId.ToString(), toCityId.ToString() };
            string redisKey = this.GetRedisKey(eVars);
            using (var client = Redis.GetClient())
            {
                busesIterator = client.ScanAllHashEntries(redisKey);
            }
            return busesIterator;
        }

Let me change this code to read before disposing the client and check if the issue still persists

10

Thanks. It worked.

The Issue was I was using iterator outside of using clause. I guess client would been disposed/assigned to some other thread. But when iterator tries to read data, it fails to read throwing above mentioned error.

Thanks a lot again for kind support

1 Like