Hi Demis,
My infrastructure:
Only my servers (that is the Redis clients) are running on Docker. The Redis servers do not. So there is no NAT issue. Also this time I think it is NOT a firewall issue, firewalls are binary: they block or they do not, but I have never seen firewalls which block randomly… So since I do get some events from various channels, it must be something else.
My configuration:
There is a lot of recommendations in the internet (and books I have as well) NOT to run multiple databases in ONE Redis instance even if it is possible. Instead you should run one Redis instance per database and assign different ports to them. Thats what I am doing. Theoretically you can have up to 16 databases per Redis instance, they are numbered 0 - 15 and you can configure thatn in redis.conf files. This blog post explains some aspects why it is NOT recommended. So as you can see in my previous shell dump, I am running six Redis instances (one for every database I have) on every server. Each instance has also a Sentinel instance which is monitoring the database. This setup is spread over four CentOS VMs which form together the ReplicaSet. Since we replicate data to another location to have only data geo-redundant (not the applications) we use for both Redis and MongoDB a ReplicaSet configuration that makes sure, that machines on the redundant site NEVER can become MASTER. In fact, they just keep an off-site database copy.
Redis PUB/SUB
I am not familiar with Redis PUB/SUB so I can’t say anything about it. I also don’t know, WHAT exactly a channel is and HOW you subscribe to them. But it looks like ALL MESSAGES in the Sentinel Log should be received by your handler. So if you compare the sentinel log with my application log, you can see what I get und what I miss. Please note, I am just looking at messages sent by the Redis instance running on Port 6387
-- SENTINEL LOG
6767:X 14 Jun 2019 10:10:22.282 # +new-epoch 8
6767:X 14 Jun 2019 10:10:22.289 # +vote-for-leader 75d6da2fe2b67695af9e015c8fc39766a02e7a41 8
6767:X 14 Jun 2019 10:10:23.252 # +odown master redis_6387_config 172.16.63.51 6387 #quorum 2/2
6767:X 14 Jun 2019 10:10:23.252 # Next failover delay: I will not start a failover before Fri Jun 14 10:10:42 2019
6767:X 14 Jun 2019 10:10:42.528 # +new-epoch 9
6767:X 14 Jun 2019 10:10:42.533 # +vote-for-leader 75d6da2fe2b67695af9e015c8fc39766a02e7a41 9
6767:X 14 Jun 2019 10:10:42.555 # Next failover delay: I will not start a failover before Fri Jun 14 10:11:02 2019
6767:X 14 Jun 2019 10:11:03.010 # +new-epoch 10
6767:X 14 Jun 2019 10:11:03.010 # +try-failover master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.016 # +vote-for-leader 547b4d726635832d950a4af6dae166889c71c0f1 10
6767:X 14 Jun 2019 10:11:03.026 # 75d6da2fe2b67695af9e015c8fc39766a02e7a41 voted for 547b4d726635832d950a4af6dae166889c71c0f1 10
6767:X 14 Jun 2019 10:11:03.028 # cb7a3251ac116a0381b70622a9433ee72bc86d34 voted for 547b4d726635832d950a4af6dae166889c71c0f1 10
6767:X 14 Jun 2019 10:11:03.072 # +elected-leader master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.072 # +failover-state-select-slave master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.155 # +selected-slave slave 172.16.63.52:6387 172.16.63.52 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.156 * +failover-state-send-slaveof-noone slave 172.16.63.52:6387 172.16.63.52 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.247 * +failover-state-wait-promotion slave 172.16.63.52:6387 172.16.63.52 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.443 # +promoted-slave slave 172.16.63.52:6387 172.16.63.52 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.443 # +failover-state-reconf-slaves master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:03.510 * +slave-reconf-sent slave 172.16.63.57:6387 172.16.63.57 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:04.147 # -odown master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:04.203 * +slave-reconf-inprog slave 172.16.63.57:6387 172.16.63.57 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:05.268 * +slave-reconf-done slave 172.16.63.57:6387 172.16.63.57 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:05.329 * +slave-reconf-sent slave 172.16.63.53:6387 172.16.63.53 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:06.301 * +slave-reconf-inprog slave 172.16.63.53:6387 172.16.63.53 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:06.301 * +slave-reconf-done slave 172.16.63.53:6387 172.16.63.53 6387 @ redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:06.354 # +failover-end master redis_6387_config 172.16.63.51 6387
6767:X 14 Jun 2019 10:11:06.354 # +switch-master redis_6387_config 172.16.63.51 6387 172.16.63.52 6387
6767:X 14 Jun 2019 10:11:06.354 * +slave slave 172.16.63.57:6387 172.16.63.57 6387 @ redis_6387_config 172.16.63.52 6387
6767:X 14 Jun 2019 10:11:06.354 * +slave slave 172.16.63.53:6387 172.16.63.53 6387 @ redis_6387_config 172.16.63.52 6387
6767:X 14 Jun 2019 10:11:06.354 * +slave slave 172.16.63.51:6387 172.16.63.51 6387 @ redis_6387_config 172.16.63.52 6387
6767:X 14 Jun 2019 10:11:11.408 # +sdown slave 172.16.63.51:6387 172.16.63.51 6387 @ redis_6387_config 172.16.63.52 6387
-- APP LOG
2019-06-14 10:10:22.297 +02:00 [DBG] [BizBusLicenseServer.AppHost] [ThreadId: 6] n/a n/a (n/a) ONSENTINELMESSAGERECEIVED event received from redis_6387_config. Message: '8' Channel '+new-epoch'...
2019-06-14 10:10:22.300 +02:00 [DBG] [BizBusLicenseServer.AppHost] [ThreadId: 6] n/a n/a (n/a) ONSENTINELMESSAGERECEIVED event received from redis_6387_config. Message: '75d6da2fe2b67695af9e015c8fc39766a02e7a41 8' Channel '+vote-for-leader'...
THAT IS THE ONLY MESSAGE I GET FROM PORT 6385, THERE ARE 9 MESSAGES I GET FROM THE INSTANCE WITH PORT 6385, EVERYTHING ELSE IS MISSING!
Just speculation here: If you subscribe to messages sent by a specific machine, e.g. the MASTER and this machine goes down - how you get your subscribed messages? From which machine, port ?
redis-cli
The redis-cli tool is VERY POWERFUL. If you have a chance, connect to a redis database instance with this tool and type monitor. You will see everything that is entering or leaving that Redis instance. It shows tons of PINGs which seem to be kind of a heart-beat sent to all RS-members etc.