'No Redis Sentinels were available' on AWS to Non-AWS

netmarblemonster · January 15, 2019, 9:09am

Hello.

Here’s my setup with a single server and 2 clients:

Redis v4.0.6 servers on (A)
ServiceStack.Redis v5.4.0 on (B) and (C)

It is for testing purposes only. All the server nodes reside in a single server (A):

On (B), the program works just fine.

On (C), the same program halts with the following stacktrace:

Unhandled Exception: ServiceStack.Redis.RedisException: No Redis Sentinels were available —> ServiceStack.Redis.RedisException: Exceeded timeout of 00:00:10 —> ServiceStack.Redis.RedisRetryableException: Socket is not connected**
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte cmdWithBinaryArgs, Func1 fn, Action1 completePipelineFn, Boolean sendWithoutRead)
— End of inner exception stack trace —
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte cmdWithBinaryArgs, Func1 fn, Action1 completePipelineFn, Boolean sendWithoutRead)
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte cmdWithBinaryArgs)
at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName)
at ServiceStack.Redis.RedisSentinelWorker.GetMasterHostInternal(String masterName)
at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo()
at ServiceStack.Redis.RedisSentinel.GetRedisManager()
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()
— End of inner exception stack trace —
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()
at ServiceStack.Redis.RedisSentinel.Start()
at M.RedisSentinel.Connect(Int32 no)

Code in M.RedisSentinel.Connect():

var sentinel = new ServiceStack.Redis.RedisSentinel(sentinels, masterName);
sentinel.RedisManagerFactory = (master, slaves) => new PooledRedisClientManager(
master,
slaves,
new RedisClientManagerConfig {
MaxReadPoolSize = 200,
MaxWritePoolSize = 200,
},
null,
200,
100);
var redis = sentinel.Start();

(C) → (A) connection seems to have no problem when tested with redis-cli.exe:

PS (C)> .\redis-cli.exe -h (A) -p 50003
(A):50003>
(A):50003>
(A):50003> info
'#Server
redis_version:4.0.6
redis_git_sha1:00000000
redis_git_dirty:0
…

I tried wireshark to see what’s happening and found some weird connection tries repeating every 0.1 seconds:

No. Time Source Destination Protocol Length Info
85 2.920814 (C) AWS (A) Server TCP 66 56974 → 50003 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
93 3.049107 (C) AWS (A) Server TCP 66 56975 → 50003 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
94 3.052747 (A) Server (C) AWS TCP 66 50003 → 56974 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
95 3.052760 (C) AWS (A) Server TCP 54 56974 → 50003 [RST] Seq=1 Win=0 Len=0
98 3.149554 (C) AWS (A) Server TCP 66 56976 → 50003 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
99 3.184345 (A) Server (C) AWS TCP 66 50003 → 56975 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
100 3.184359 (C) AWS (A) Server TCP 54 56975 → 50003 [RST] Seq=1 Win=0 Len=0
102 3.260759 (C) AWS (A) Server TCP 66 56977 → 50003 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
103 3.283531 (A) Server (C) AWS TCP 66 50003 → 56976 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
104 3.283544 (C) AWS (A) Server TCP 54 56976 → 50003 [RST] Seq=1 Win=0 Len=0
105 3.394375 (A) Server (C) AWS TCP 66 50003 → 56977 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
106 3.394390 (C) AWS (A) Server TCP 54 56977 → 50003 [RST] Seq=1 Win=0 Len=0
…

Nothing like this happens when I just used ServiceStack.Redis.RedisClient(host, port) to make direct connection to its master.

mythz · January 15, 2019, 3:12pm

Note: you should have at least 3 Sentinels in any Redis configuration.

The Exception message says it can’t establish a TCP connection with the Sentinel Server instance, can you try to make a direct connection to the Sentinel Instance with ServiceStack.Redis? What port is the Sentinel instance running on? Are you connecting with redis-cli to the sentinel instance on the same instance that ServiceStack.Redis is trying to connect to it?

The connection can also be due to a firewall, make sure you have the port open for each redis instance you’re running.

netmarblemonster · January 16, 2019, 2:19am

Ports being used in (A) are:
50001 master
50002 slave
50003 sentinel
All the ports are open and listening. (Tested (C) → (A) connection with telnet + redis-cli.exe)

Yes, I’m already making direct connections to sentinel (50003).
redis-cli.exe and ServiceStack.Redis are run on the same instance (C).

As I noted ealier in wireshark output, the strange thing is that ServiceStack.Redis keeps sending many connection requests without waiting for server (A)'s second phase (ACK), and it closes connections even after received the replies (ACK) properly.

mythz · January 16, 2019, 3:32am

ServiceStack.Redis is just trying to establish a standard TCP Connection using .NET’s built-in System.Net.Sockets.Socket, it doesn’t have any control over the low-level TCP hand-shaking, that’s all handled by .NET’s internal networking classes.

I’m assuming it’s a firewall or faulty networking issue given it’s unable to establish a basic TCP Socket connection where I’m expecting no .NET Library will be able to maintain a TCP connection.

For comparison purposes I’ve (temporarily) opened up a Redis master + sentinel instance on a bare bones Amazon Linux Instance at:

52.7.181.87
redis-server: 6379
redis-sentinel: 26379 (mymaster)

And this what it looks like establishing a tcp connection + running the simple PING command:

RedisConfig.AssumeServerVersion = 4000; //disable calling INFO to infer redis version

var client = new RedisClient("52.7.181.87", 26379);

Assert.That(client.Ping());

And the Redis protocol submitted by following the TCP Stream:

This is what a basic Sentinel + Redis Manager looks like:

RedisConfig.AssumeServerVersion = 4000;

var sentinel = new RedisSentinel("52.7.181.87:26379") {
    IpAddressMap = {
        {"127.0.0.1", "52.7.181.87"}
    }
};

var redisManager = sentinel.Start();

using (var client = redisManager.GetClient())
{
    Assert.That(client.Ping());
}

Sentinel + Redis Wireshark packets:

TCP Stream (Sentinel):

TCP Stream (Redis):

mythz · January 16, 2019, 3:51am

The major difference from our packets is that your client is sending Explicit congestion notification packets which could also be a cause of the TCP Reset (RST) connections:

If the client is running on a Windows OS you can try disabling ECN to see if it helps:

$ netsh int tcp set global ecncapability=disabled

netmarblemonster · January 16, 2019, 6:37am

Tried to turn off ECN flag and it had no effect. (confirmed the removal of ECN flag using wireshark)

It only happens when the client implementing ServiceStack.Redis is run on AWS instances. I’ve tried 2 different instances and the both resulted in failure.

mythz · January 16, 2019, 6:48am

Can you try connecting to my AWS Amazon Linux Redis Sentinel instance from your client (its still publicly accessible) to see if you can connect?

var client = new RedisClient("52.7.181.87", 26379);

var info = client.SentinelMaster("mymaster");

info.PrintDump();

Also what is your client and server OS?

netmarblemonster · January 16, 2019, 8:54am

I’ve just run several times. No exception occurred.

Output:

PS C:\Users\Administrator\Desktop\Sen> .\Sen.exe
name=mymaster
ip=127.0.0.1
port=6379
runid=34c9e9b828035fbc40f64f2ee2ed28729ae7ca90
flags=master
link-pending-commands=0
link-refcount=1
last-ping-sent=0
last-ok-ping-reply=578
last-ping-reply=578
down-after-milliseconds=30000
info-refresh=6643
role-reported=master
role-reported-time=21398672
config-epoch=0
num-slaves=0
num-other-sentinels=0
quorum=2
failover-timeout=180000
parallel-syncs=1
PS C:\Users\Administrator\Desktop\Sen>

This code is working too:

var sentinel = new RedisSentinel(“52.7.181.87:26379”) {
IpAddressMap = { { “127.0.0.1”, “52.7.181.87” } }
};

Console.WriteLine(“Start”);
var redisManager = sentinel.Start();

Console.WriteLine(“GetClient”);
using (var client = redisManager.GetClient()) {

Console.WriteLine(“Ping”);
client.Ping();
}
Console.WriteLine(“Yay”);

Output (AWS → AWS):

Start
GetClient
Ping
Yay

Once I replaced target address to company local’s which is not on AWS, it fails. (Ports are properly open.)
Output (AWS → non-AWS):

Start

Unhandled Exception: ServiceStack.Redis.RedisException: No Redis Sentinels were available —> ServiceStack.Redis.RedisE
xception: Exceeded timeout of 00:00:10 —> ServiceStack.Redis.RedisRetryableException: Socket is not connected
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte cmdWithBinaryArgs, Func1 fn, Action1 completePipeli
neFn, Boolean sendWithoutRead)
— End of inner exception stack trace —
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte cmdWithBinaryArgs, Func1 fn, Action1 completePipeli
neFn, Boolean sendWithoutRead)
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte cmdWithBinaryArgs)
at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName)
at ServiceStack.Redis.RedisSentinelWorker.GetMasterHostInternal(String masterName)
at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo()
at ServiceStack.Redis.RedisSentinel.GetRedisManager()
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()
— End of inner exception stack trace —
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()
at ServiceStack.Redis.RedisSentinel.Start()
at Sen.Program.Main(String args)

Seems like it’s working only if the nodes on the both sides are AWS instances?

mythz · January 16, 2019, 2:15pm

It works in all cases, when both are on AWS, when only the server is, when only the client is calling a Redis server on my hetzner server in Germany.

I’m unable to repro this issue at all which seems to be an issue with calling your Redis server, which I still assume is the fault of a firewall or faulty network condition.