How to make RedisManagerPool close idle connections

ServiceStack.Redis
7

No the connection returns to the pool as soon as you dispose of it. i.e:

using (var redis = redisManager.GetClient())
{
}

The IdleTimeoout is how long period of inactivity before the redisManager considers the connection to be stale and then discarded. It should be a little lower than the redis-server configured TCP keepalive (default 300 seconds), but otherwise there’s no reason that it should be set so low.

8

hmm
Part of my Redis.conf:

# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
#    equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
tcp-keepalive 0

I guess I should set it to 300, right?

9

Or mb 0 means 300 as default value?

10

Not configuring it at all leaves it as the default of 240 seconds.

The 300 seconds is the default for the Redis Server, not the client.

11 
class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("First query done");
            Console.ReadLine();

            //press ENTER after 5 mins


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();

        }
    }

When I launch this console app on my local PC it works fine
On Azure VM (Windows, B2S) same console app throws an exception

To catch it I have to wait 5 mins after the first query, press ENTER to make console app send second query
Any ideas? Maybe there are some inbound / outbound configs on windows which I have to edit?

12

13

This error is from Automatic Retries that are unable to establish a TCP connection within the default RedisConfig.DefaultRetryTimeout of 10 seconds.

Can you try using a new RedisManagerPool to check whether its an issue with the existing open connection or just not being able to connect to Redis Azure.

 redisManager = new RedisManagerPool("redis://xxx:6379?db=2");
//press ENTER after 5 mins
using (var red = redisManager.GetClient())
{
    string val = red.Get<string>("samplekey");
}
Console.WriteLine("Second query done");
Console.ReadLine();
14

Got it, I’ll check it tomorrow and will let you know. Thank you very much for your help

P.S. I don’t use Redis Azure. My redis instance is Ubuntu 16.04 droplet on Digital Ocean

15

Hello
Tried to run console app with this code

class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }

            Console.WriteLine("First query done");
            Console.ReadLine();

            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();
        }

It works fine but I get 2 clients in client list:

id=5 addr=yyy:62361 fd=9 name= age=311 idle=311 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=6 addr=yyy:62365 fd=10 name= age=7 idle=6 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

netstat cmd:

tcp        0      0 xxx:6379      yyy:62365      ESTABLISHED 7740/redis-server *
tcp        0      0 xxx:6379      yyy:62361      ESTABLISHED 7740/redis-server *

So, is this behaviour correct?

16

Yeah that all looks fine from here.

17

Good, thanks

But I don’t really understand how to use redisManager reassignment in my case
There is singleton instance of RedisManagerPool in production (in docs you say that it is thread-safe manager, so it may be used as singleton instance)
I’m using ServiceStack.Redis in my game server and redisManager.GetClient() is called everytime the game finishes.
Should I reassign redisManager after some period of time or there are better options to handle this issue?

18

UPD:
Sometimes this code

class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("First query done");
            Console.ReadLine();

            //press ENTER after 5 mins


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();

        }
    }

causes exception even on my local PC, so it doesn’t look like that the problem is reproducible only on Azure Windows VM

19

I’ve tried running this multiple times and was unable to repro the issue, then got sick of waiting for 5 mins each run so I rewrote it in a loop which reruns the code after 6 minutes:

using (var redis = redisManager.GetClient())
{
    redis.Set("samplekey", "value");
}
    
try
{
    var i = 0;
    while (true)
    {
        using (var redis = redisManager.GetClient())
        {
            string val = redis.Get<string>("samplekey");
            $$"{DateTime.Now.ToShortTimeString()} {++i} query done: {val}".Print();
        }

        "Waiting for 6 minutes...".Print();
        Thread.Sleep(TimeSpan.FromMinutes(6));
    }
}
catch (Exception e)
{
    Console.WriteLine(e);
    throw;
}

So far the output is:

3:58 PM 1 query done: value
Waiting for 6 minutes...
4:04 PM 2 query done: value
Waiting for 6 minutes...
4:10 PM 3 query done: value
Waiting for 6 minutes...
4:16 PM 4 query done: value
Waiting for 6 minutes...
4:22 PM 5 query done: value
Waiting for 6 minutes...
4:28 PM 6 query done: value
Waiting for 6 minutes...

Going to leave it running for another hour to see if I can repro the issue.

20

Ran for another hour without a repro:

4:32 PM 1 query done: value
Waiting for 6 minutes...
4:38 PM 2 query done: value
Waiting for 6 minutes...
4:44 PM 3 query done: value
Waiting for 6 minutes...
4:50 PM 4 query done: value
Waiting for 6 minutes...
4:56 PM 5 query done: value
Waiting for 6 minutes...
5:02 PM 6 query done: value
Waiting for 6 minutes...
5:08 PM 7 query done: value
Waiting for 6 minutes...
5:14 PM 8 query done: value
Waiting for 6 minutes...
5:20 PM 9 query done: value
Waiting for 6 minutes...
5:26 PM 10 query done: value
Waiting for 6 minutes...

Going to have to stop it there because I need to stop all Apps and close or IDEs to clear my local NuGet package cache.

21

Could you please share your redis.conf file? Would be much appreciated

I start thinking that the problem can be caused by bad configs on my redis instance

Thanks for your help

22

I’m just using the default redis config file, unchanged from installation.

23

So do I
Honestly, I don’t really know what else can be the source of my problem
Any ideas?

24

Can’t really say without being able to repro it, I can try running it again for a bit longer.

25

Still nothing:

8:12 PM 1 query done: value
Waiting for 6 minutes...
8:18 PM 2 query done: value
Waiting for 6 minutes...
8:24 PM 3 query done: value
Waiting for 6 minutes...
8:30 PM 4 query done: value
Waiting for 6 minutes...
8:36 PM 5 query done: value
Waiting for 6 minutes...
8:42 PM 6 query done: value
Waiting for 6 minutes...
8:48 PM 7 query done: value
Waiting for 6 minutes...
8:54 PM 8 query done: value
Waiting for 6 minutes...
9:00 PM 9 query done: value
Waiting for 6 minutes...
9:06 PM 10 query done: value
Waiting for 6 minutes...
9:12 PM 11 query done: value
Waiting for 6 minutes...
9:18 PM 12 query done: value
Waiting for 6 minutes...
9:24 PM 13 query done: value
Waiting for 6 minutes...
9:30 PM 14 query done: value
Waiting for 6 minutes...

I’ve started running it against one of my redis-servers across the world in Germany to see if I can get a repro…

26

Ran it for another 1.5 hours against a redis-server instance in Germany with still no issue:

9:31 PM 1 query done: value
Waiting for 6 minutes...
9:37 PM 2 query done:
Waiting for 6 minutes...
9:43 PM 3 query done:
Waiting for 6 minutes...
9:49 PM 4 query done:
Waiting for 6 minutes...
9:55 PM 5 query done:
Waiting for 6 minutes...
10:01 PM 6 query done:
Waiting for 6 minutes...
10:07 PM 7 query done:
Waiting for 6 minutes...
10:13 PM 8 query done:
Waiting for 6 minutes...
10:19 PM 9 query done:
Waiting for 6 minutes...
10:25 PM 10 query done:
Waiting for 6 minutes...
10:31 PM 11 query done:
Waiting for 6 minutes...
10:37 PM 12 query done:
Waiting for 6 minutes...
10:43 PM 13 query done:
Waiting for 6 minutes...
10:49 PM 14 query done:
Waiting for 6 minutes...
10:55 PM 15 query done:
Waiting for 6 minutes...
11:01 PM 16 query done:
Waiting for 6 minutes...

Given it was unable to establish a TCP connection in your last stack trace it’s possible your network or configuration could be faulty. It indicates the Redis Client tried to reconnect up to the default RedisConfig.DefaultRetryTimeout before giving up and throwing.

You can try extending RedisConfig.DefaultRetryTimeout for a longer time or disable connection pooling by using BasicRedisClientManager instead of RedisManagerPool.