Hello, I am hoping I will be able to get some help here. Allow me to first explain the situation. We have three developers, one working in US and two in UK. The servers are located in US and the UK developers use VPN to access the US network. Now, the two UK developers have their own office where everything works as expected. When they take their computers home, using the same VPN and the same code, one of them is having connectivity issues. This situation could be explained by home network firewall blocking connections from the .Net application to the US server via VPN. However, when connected via 4G hotspot the same issue occurs. Moreover, when using redis-cli the developer is able to connect from his home on the same computer to all sentinels without problems. Also, instead of connecting to Sentinel, we tried connecting to Redis instance directly and that portion worked. The error is the following:
Unhandled exception. ServiceStack.Redis.RedisException: No Redis Sentinels were available
---> ServiceStack.Redis.RedisException: Exceeded timeout of 00:00:10
---> ServiceStack.Redis.RedisRetryableException: Socket is not connected
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 605
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 658
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 733
at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient.cs:line 1551
at ServiceStack.Redis.RedisSentinelWorker.GetMasterHostInternal(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 106
at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 75
at ServiceStack.Redis.RedisSentinel.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 418
at ServiceStack.Redis.RedisSentinel.GetRedisManager() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 292
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 312
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 330
at ServiceStack.Redis.RedisSentinel.Start() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 174
at
mythz
March 17, 2020, 3:19am
2
This shows the Auto Retry Limit for trying to resolve a Redis Sentinel Host was exceeded, i.e. you have more networking connectivity issues than just latency, this StackTrace suggests it wasn’t able to establish a TCP connection within 10 seconds.
You can increase the Auto Retry limit with:
RedisConfig.DefaultRetryTimeout = 30 * 1000; // e.g. 30 seconds
Alternatively as it was unable to establish a connection to the sentinel host provided to be able to discover the other sentinels, you can try specifying all sentinel hosts manually, e.g:
var sentinelHosts = new[]{ "sentinel1", "sentinel2:6390", "sentinel3" };
var sentinel = new RedisSentinel(sentinelHosts, masterName: "mymaster");
I tried specifying them manually and I set ScanForOtherSentinels = false; and I get the same error.
Also, I tried pinging a sentinel host like this:
var client = new RedisClient("debian-dev01", 26379);
client.Ping();
System.Console.WriteLine("Pinged");
System.Console.ReadLine();
And it worked okay. But when I do this:
var client = new RedisSentinel("debian-dev01:26379", "devredis");
client.Start();
System.Console.WriteLine("Started");
System.Console.ReadLine();
I get the timeout error above. Anything else we can try?
mythz
March 17, 2020, 3:49pm
4
Can you try enabling logging:
LogManager.LogFactory = new ConsoleLogFactory(debugEnabled:true);
If that doesn’t help identify the issue you can enable verbose logging with:
RedisConfig.EnableVerboseLogging = true;
We enabled logging:
DEBUG: Set up Redis Sentinel on debian-dev01:26380
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
DEBUG: #3 Socket Connect failed
DEBUG: #3 SendReceive Exception: Socket is not connected
ERROR: #3 Exceeded timeout of 00:00:03
DEBUG: Set up Redis Sentinel on debian-dev01:26380
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Socket Connect failed
DEBUG: #4 SendReceive Exception: Socket is not connected
ERROR: #4 Exceeded timeout of 00:00:03
DEBUG: Set up Redis Sentinel on debian-dev01:26380
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Socket Connect failed
DEBUG: #5 SendReceive Exception: Socket is not connected
ERROR: #5 Exceeded timeout of 00:00:03
Unhandled exception. ServiceStack.Redis.RedisException: No Redis Sentinels were available
---> ServiceStack.Redis.RedisException: Exceeded timeout of 00:00:03
---> ServiceStack.Redis.RedisRetryableException: Socket is not connected
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 605
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 658
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 733
at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient.cs:line 1551
at ServiceStack.Redis.RedisSentinelWorker.GetMasterHostInternal(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 106
at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 75
at ServiceStack.Redis.RedisSentinel.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 418
at ServiceStack.Redis.RedisSentinel.GetRedisManager() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 292
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 312
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 330
at ServiceStack.Redis.RedisSentinel.Start() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 174
at TestClient.Console.Program.Main(String[] args) in
mythz
March 17, 2020, 4:06pm
6
hubert.jarema.dm:
debian-dev01:26380
It’s unable to establish a TCP connection to debian-dev01:26380 where your test was for debian-dev01:26379.
We have two 26379 and 26380. Both are failing.
mythz
March 17, 2020, 4:33pm
8
Don’t know then, it says it can’t establish a TCP connection, there’s nothing that can be done if it can’t make a TCP connection. A packet sniffer like WireShark will let you see the individual SYN/ACK TCP/IP packets that’s used to establish a TCP connection, app firewalls will prevent these from being sent/received.
Can you try running this Simple App to simulate what the Client is doing?
var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect("debian-dev01", 26380);
Console.WriteLine(socket.Connected ? "CONNECTED!" : "FAILED");
I know, it’s really strange, especially since we can call RedisClient().Ping() to each of those sentinel servers without errors. The code for the Socket above also worked. We received Connected = true;
mythz
March 17, 2020, 4:57pm
10
I’ll look to see if I can add better logging to see if it can better identify what’s going on…
1 Like
mythz
March 17, 2020, 5:43pm
11
Can you try downloading the latest v5.8.1 on MyGet and see if the additional logging yields any more clues.
Here is what we have now:
DEBUG: Attempt to connect to next sentinel 'debian-dev01:26380'...
DEBUG: Set up Redis Sentinel on debian-dev01:26380
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
DEBUG: #4 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #4 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #4 SendReceive Exception: Socket is not connected
ERROR: #4 Exceeded timeout of 00:00:03
DEBUG: Failed to GetRedisManager(): Exceeded timeout of 00:00:03
DEBUG: Attempt to connect to next sentinel 'debian-dev01:26380'...
DEBUG: Set up Redis Sentinel on debian-dev01:26380
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
DEBUG: #5 Attempting async connection to 'debian-dev01:26380' (SEND 100, RECV 100 Timeouts)...
DEBUG: #5 Socket failed connect to 'debian-dev01:26380' (ConnectTimeout 100)
DEBUG: #5 SendReceive Exception: Socket is not connected
ERROR: #5 Exceeded timeout of 00:00:03
DEBUG: Failed to GetRedisManager(): Exceeded timeout of 00:00:03
Unhandled exception. ServiceStack.Redis.RedisException: No Redis Sentinels were available
---> ServiceStack.Redis.RedisException: Exceeded timeout of 00:00:03
---> ServiceStack.Redis.RedisRetryableException: Socket is not connected
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 611
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 664
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 739
at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient.cs:line 1551
at ServiceStack.Redis.RedisSentinelWorker.GetMasterHostInternal(String masterName) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 106
at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinelWorker.cs:line 75
at ServiceStack.Redis.RedisSentinel.GetSentinelInfo() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 434
at ServiceStack.Redis.RedisSentinel.GetRedisManager() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 293
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 314
--- End of inner exception stack trace ---
at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 343
at ServiceStack.Redis.RedisSentinel.Start() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisSentinel.cs:line 176
at TestClient.Console.Program.Main(String[] args) in
mythz
March 17, 2020, 6:18pm
13
Ok so it looks like you’re using non-default configuration parameters. Can you please try again just using the defaults? These custom timeouts (in ms) are too low if you’re trying to connect over mobile internet & VPN’s.
Ok that would make sense. I don’t have any custom settings though. Can there be something else setting it? Is that a Redis server setting? This is the entirety of my code:
LogManager.LogFactory = new ConsoleLogFactory(debugEnabled: true);
RedisConfig.EnableVerboseLogging = true;
var client = new RedisSentinel("debian-dev01:26380", "devredis");
client.ScanForOtherSentinels = false;
client.Start();
System.Console.WriteLine("Started");
System.Console.ReadLine();
mythz
March 17, 2020, 6:34pm
15
Are you using RedisConfig anywhere in your code-base? Look for 100 timeouts.
mythz
March 17, 2020, 6:37pm
16
Actually it looks like their RedisSentinel timeouts, can you try:
var client = new RedisSentinel("debian-dev01:26380", "devredis") {
SentinelWorkerConnectTimeoutMs = -1,
SentinelWorkerReceiveTimeoutMs = -1,
SentinelWorkerSendTimeoutMs = -1,
}
I’ll increase the defaults to 250ms…
Yes, that worked. I’ll change my code to 250ms for now as well, and we’ll do some more testing. Thanks for your help! Much appreciated!
1 Like
mythz
March 17, 2020, 7:06pm
18
Great happy to hear that resolved the issue! An updated RedisSentinel with 250ms defaults is now available on MyGet , you’ll need to clear your NuGet packages cache to download the latest v5.8.1 release.
1 Like