Migrate from Redis standalone to Redis.Sentinel

tbednarz · March 22, 2019, 10:14am

We are moving our Redis stuff to a more robust setup with a ReplicaSet consisting of 1 master and two slaves.

The ReplicaSet is running on CentOS and working fine, switching slaves to a master etc works fine. Now I need to migrate my existing application code which is currently only supporting ‘standalone’ Redis servers to support the new Redis replica set with sentinel.

Currently I use in my AppHost.Configure() code like so: (Sorry, don’t know how to escape the ‘$’ in C# code…)

connStrCfg = $"redis://{redisServer}:{redisPort}?ConnectTimeout=5000&Db={BbConsumerConstants.RedisProdDb}&password={HttpUtility.UrlEncode(pwdRedisConfig)}";

var redisCfgPool = new PooledRedisClientManager(connStrCfg);
container.Register(c => new SysConfigRepository(redisCfgPool));

I am reading the Wiki documentation and have a few questions:

The documentation states:

var sentinel = new RedisSentinel(sentinelHosts, masterName: "mymaster");

This is a bit confusing! I guess with masterName you do not mean the name of the machine which is master but you mean the replica set name? The name of the master can change whenever it goes down and a slave becomes master. How can I now which machine is currently master when I (re-)start my application?? The redis sentinel documentation says:

sentinel monitor mymaster 127.0.0.1 6379 2
# where the meaning is as follows
sentinel monitor <master-group-name> <ip> <port> <quorum>

This is what you have to setup in the senitel.conf file on Linux. I call master-group-name ‘Redis replica set name’ which indicates a bit more clear what it means. So here is an extract of my sentinel.conf on CentOS:

sentinel redis-config-rs dbinfra2.tbhome.int 6381 2
sentinel auth-pass redis-config-rs MySuperDuperTopSecretPassword
sentinel down-after-milliseconds redis-config-rs 15000
sentinel parallel-syncs redis-config-rs 1
sentinel failover-timeout redis-config-rs 180000

Where redis-config-rs is the name of the replica set, a logical name of a group of hosts that make up the replica set (you name this myMaster). This appears in many of the config settings for sentinel to define to which replica set a value belongs.

So here is some of my ‘migrated’ code to support sentinel: Is this correct???

// replsetname would be  "redis-config-rs" from above config file
var sentinel = new RedisSentinel(sentinelHosts, masterName: replSetName);

Other questions

I have three nodes in my replica set, apart from the hostname all parameters are the same. Do I have to define three sentinel.HostFilter like so?

sentinel.HostFilter = host1 => $"{host1}?Db={BbConsumerConstants.RedisProdDb}&RetryTimeout=5000&password={HttpUtility.UrlEncode(pwdRedisConfig)}";
sentinel.HostFilter = host2 => $"{host2}?Db={BbConsumerConstants.RedisProdDb}&RetryTimeout=5000&password={HttpUtility.UrlEncode(pwdRedisConfig)}";
sentinel.HostFilter = host3 => $"{host3}?Db={BbConsumerConstants.RedisProdDb}&RetryTimeout=5000&password={HttpUtility.UrlEncode(pwdRedisConfig)}";
In the documentation you write that there is a newer Pool class RedisManagerPool. What are the differences to PooledRedisClientManager and should I use the newer one?
Tricky other configuration properties. Do I need to implement any of these? What is the behavior if the master is rebooted and I try to write something to Redis? Will it simply fail or can I handle it in a way? In my configuration (see above) I configured to wait 15 seconds until a new Master is elected. I guess it is much to long, but the final value I will use depends on many things, also on how I configure the service stack client.

Thanks for some hints on these points.

mythz · March 22, 2019, 4:56pm

FYI you need to use double ‘$$’ to escape in discourse.

The Redis Configuring Sentinel docs explains the usage of mymaster for monitoring different groups of Redis master/slave instances, mymaster is the default name, if you only need to monitor 1 redis master/slaves configuration just leave it as the default. I don’t know if this is what you mean by replica set name or not.

There’s only a single HostFilter which is used to change the redis connection string RedisSentinel when it connects to the Redis master instance, there can only be 1 configuration.

The differences between the different Redis Client Managers is explained on the project home page.

No they all have defaults.

The sentinel detects the master is down and coordinates promoting one of the slaves to master, read this to find out more info on what happens when a Redis Slave or Redis Master is failing.

When a command fails with ServiceStack.Redis it automatically retries up to the RetryTimeout (default 10s).

tbednarz · March 25, 2019, 9:01am

Hi Demis,
Thanks for that information. First I got exceptions but now it looks like I can connect. Figured out, that the guy who wrote the ANSIBLE scripts to automate roll out of redis and all my config files changed the name of the replica set name.

Yes, MyMaster is the default but it is a little bit like Hello World and rarely suitable in production. I run six databases, each on a different port in a replica set of currently three nodes. I name these replica sets based on database content, eg ‘cache’ or ‘configuration’ etc.

I will have to test failover behaviour to fine tune the parameters in my sentinel.conf files. The sentinels modify the config files when a master switches, and the nasty thing is, that it replaces DNS names with IP addresses. Here is a sample from one of my files found in the /etc folder of my LINUX box:

# Generated by CONFIG REWRITE
maxclients 4064
supervised systemd
sentinel leader-epoch redis_6381_config 0
sentinel known-replica redis_6381_config 172.16.63.179 6381
sentinel known-replica redis_6381_config 172.16.63.53 6381
sentinel known-sentinel redis_6381_config 172.16.63.53 26381 0e91e3cd0d8009afb233f2f4dfc867b2b70f01e0
sentinel known-sentinel redis_6381_config 172.16.63.179 26381 5bf7cd009a4ac9911f114ef27bfbf866c3a21c3a
sentinel current-epoch 0

This is what is shown when you execute info replication using the redis-cli.

Funny, it looks like redis uses config files stored under /etc as ‘system database’. My sys admins do not like that at all!

tbednarz · March 25, 2019, 1:56pm

Inserts are INCREDIBLY SLOW, two to three seconds for ONE insert. Before Sentinel I did around several hundred per second! Code to insert:

public void SetConfigurationItem(SysConfigItem sysConfig)
{
	Logger.Debug($$"Storing item to config database on Redis. Item: {sysConfig.ToJson()}");
	try
	{
		using (var redis = RedisManager.GetClient())
		{
			redis.SetEntryInHash(sysConfig.Key.ToLower(), sysConfig.ValueName, sysConfig.Value);
		}
	}
	catch (Exception e)
	{
		Logger.Error($$"Exception when creating new configuration item {sysConfig}. Exception: {e}");
		throw;
	}
	Logger.Debug("Success.");
}

Logger output:

2019-03-25 15:01:26.513 +01:00 [DBG] [BizBusOperationsManagerServer.ServiceInterface.Repositories.SysConfigRepository] [ThreadId 1] Storing key cfg:bizbussys:servicehosts, ValueName: bbdockerhost, Value: linuxdev-tbws2 to config database on Redis
2019-03-25 15:01:29.298 +01:00 [DBG] [BizBusOperationsManagerServer.ServiceInterface.Repositories.SysConfigRepository] [ThreadId 1] Success.
2019-03-25 15:01:31.916 +01:00 [DBG] [BizBusOperationsManagerServer.ServiceInterface.Repositories.SysConfigRepository] [ThreadId 1] Storing key cfg:bizbussys:serviceports, ValueName: bbinvoice, Value: 6081 to config database on Redis
2019-03-25 15:01:33.660 +01:00 [DBG] [BizBusOperationsManagerServer.ServiceInterface.Repositories.SysConfigRepository] [ThreadId 1] Success.

As you can see, it takes more than two seconds for one insert. Configuration problem??

If I use “Redis Desktop Manager” GUI from Windows, inserts are done instantly…

mythz · March 25, 2019, 3:13pm

That is very slow, is it just the first time the client is retrieved from the pool or is that the avg time for inserts?

If you provide a stand-alone console App I can run, I’ll let you know what I get when I run it against my Redis sentinel configuration.

tbednarz · March 25, 2019, 4:11pm

It is for EVERY insert / update. I will try a batch insert with redis-cli and see how this performs, just to be sure I do not have a problem with my VMs and network. I’ll let you know what the result is and then need to see, if I can extract some code…

tbednarz · March 25, 2019, 5:15pm

Ok, my Redis infrastructure is definitely ultra fast. I did some stresstest using redis-cli. In a bash you can add the following commands:

mkdir ~/redistest
for N in $(seq 1 500000); do echo "SADD test $N"; done > bulk-data-insert.txt
# use the --pipe parameter for bulk insert, e.g.
cat ~/redistest/bulk-data-insert.txt | redis-cli -h <host> -p <port> -a <your password if you need one> --pipe

Replace the values between <> with what your system requires and make sure your host is a MASTER.

It took about THREE seconds to generate the txt file on the local machine and it is quite BIG
-rw-r--r-- 1 root root 8388895 Mar 25 18:07 bulk-data-insert.txt
It took about ONE second to write the stuff in the LOCAL redis database and about THREE seconds to write those records over the LAN on a different redis (master) server. Only a few seconds later (less than 10) all those records where visible on my two slaves.

So I will see if I can write a console app which lets you reproduce it… I’ll let you know tomorrow.

mythz · March 25, 2019, 11:42pm

ok but know that bulk inserts isn’t going to be comparable to resolving open connections + issuing single commands each time. If you want to do bulk inserts with ServiceStack.Redis you would combine all operations into a single Transaction or Pipeline request where it’s sent together.