How to make RedisManagerPool close idle connections

Problem:
Client from RedisManagerPool.GetClient() makes query:

using(IRedisClient client = new manager.GetClient()){
    //query
}

5 mins later I try to send another query and I get an exception:
Exceeded timeout of 00:00:00.0300000
As I noticed, there is an idle connection to redis instance which may cause some troubles
How can I handle this connection?
Connection string:

redisManagerPool = new RedisManagerPool(
                    string.Format(
                        "{0}:{1}?db={2}&idletimeoutsecs=4", 
                        redisConnection.host, 
                        redisConnection.port, 
                        redisConnection.db
                    )
                );

I also tried to set IdleTimeOutSecs but it didn’t help me
Would be appreciated for any help.
Thanks!

P.S Tell me if you need some additional info about issue

How can this issue be reproduced?

Can you also provide the full StackTrace of the Exception.

Why are you setting the IdleTimeout so low? It defaults to 240 secs.

I set IdleTimeout to 4 for tests
There is similar issue when I leave this value default
Exception:

2019-05-15 15:33:42,375 [14] ERROR IronHeadGames.Leaderboard.LeaderboardApi - CheckUsername exception: Exceeded timeout of 00:00:00.0300000 
 Stack:    at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisClient.GetValueFromHash(String hashId, String key)
   at IronHeadGames.Leaderboard.LeaderboardApi.CheckUsername(Int32 playerId, String username, String game)

Do you need console client? If so I’ll try to reproduce it later and post project on github

Just need anything I can run locally to repro the issue.

Got it
I’ll post here results

Just to be sure

Does this value mean that opened connect will return to pool after 240 secs(default value) and I won’t see any clients on redis instance through CLIENT LIST command?

No the connection returns to the pool as soon as you dispose of it. i.e:

using (var redis = redisManager.GetClient())
{
} 

The IdleTimeoout is how long period of inactivity before the redisManager considers the connection to be stale and then discarded. It should be a little lower than the redis-server configured TCP keepalive (default 300 seconds), but otherwise there’s no reason that it should be set so low.

hmm
Part of my Redis.conf:

# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
#    equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
tcp-keepalive 0

I guess I should set it to 300, right?

Or mb 0 means 300 as default value?

Not configuring it at all leaves it as the default of 240 seconds.

The 300 seconds is the default for the Redis Server, not the client.

class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("First query done");
            Console.ReadLine();

            //press ENTER after 5 mins


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();

        }
    }

When I launch this console app on my local PC it works fine
On Azure VM (Windows, B2S) same console app throws an exception

To catch it I have to wait 5 mins after the first query, press ENTER to make console app send second query
Any ideas? Maybe there are some inbound / outbound configs on windows which I have to edit?

This error is from Automatic Retries that are unable to establish a TCP connection within the default RedisConfig.DefaultRetryTimeout of 10 seconds.

Can you try using a new RedisManagerPool to check whether its an issue with the existing open connection or just not being able to connect to Redis Azure.

 redisManager = new RedisManagerPool("redis://xxx:6379?db=2");
//press ENTER after 5 mins
using (var red = redisManager.GetClient())
{
    string val = red.Get<string>("samplekey");
}
Console.WriteLine("Second query done");
Console.ReadLine();

Got it, I’ll check it tomorrow and will let you know. Thank you very much for your help

P.S. I don’t use Redis Azure. My redis instance is Ubuntu 16.04 droplet on Digital Ocean

Hello
Tried to run console app with this code

class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }

            Console.WriteLine("First query done");
            Console.ReadLine();

            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();
        }

It works fine but I get 2 clients in client list:

id=5 addr=yyy:62361 fd=9 name= age=311 idle=311 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=6 addr=yyy:62365 fd=10 name= age=7 idle=6 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

netstat cmd:

tcp        0      0 xxx:6379      yyy:62365      ESTABLISHED 7740/redis-server *
tcp        0      0 xxx:6379      yyy:62361      ESTABLISHED 7740/redis-server *

So, is this behaviour correct?

Yeah that all looks fine from here.

Good, thanks

But I don’t really understand how to use redisManager reassignment in my case
There is singleton instance of RedisManagerPool in production (in docs you say that it is thread-safe manager, so it may be used as singleton instance)
I’m using ServiceStack.Redis in my game server and redisManager.GetClient() is called everytime the game finishes.
Should I reassign redisManager after some period of time or there are better options to handle this issue?

UPD:
Sometimes this code

class Program
    {
        private static RedisManagerPool redisManager;

        static void Main(string[] args)
        {
            redisManager = new RedisManagerPool("redis://xxx:6379?db=2");

            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("First query done");
            Console.ReadLine();

            //press ENTER after 5 mins


            using (var red = redisManager.GetClient())
            {
                string val = red.Get<string>("samplekey");
            }
            Console.WriteLine("Second query done");
            Console.ReadLine();

        }
    }

causes exception even on my local PC, so it doesn’t look like that the problem is reproducible only on Azure Windows VM

I’ve tried running this multiple times and was unable to repro the issue, then got sick of waiting for 5 mins each run so I rewrote it in a loop which reruns the code after 6 minutes:

using (var redis = redisManager.GetClient())
{
    redis.Set("samplekey", "value");
}
    
try
{
    var i = 0;
    while (true)
    {
        using (var redis = redisManager.GetClient())
        {
            string val = redis.Get<string>("samplekey");
            $$"{DateTime.Now.ToShortTimeString()} {++i} query done: {val}".Print();
        }

        "Waiting for 6 minutes...".Print();
        Thread.Sleep(TimeSpan.FromMinutes(6));
    }
}
catch (Exception e)
{
    Console.WriteLine(e);
    throw;
}

So far the output is:

3:58 PM 1 query done: value
Waiting for 6 minutes...
4:04 PM 2 query done: value
Waiting for 6 minutes...
4:10 PM 3 query done: value
Waiting for 6 minutes...
4:16 PM 4 query done: value
Waiting for 6 minutes...
4:22 PM 5 query done: value
Waiting for 6 minutes...
4:28 PM 6 query done: value
Waiting for 6 minutes...

Going to leave it running for another hour to see if I can repro the issue.

Ran for another hour without a repro:

4:32 PM 1 query done: value
Waiting for 6 minutes...
4:38 PM 2 query done: value
Waiting for 6 minutes...
4:44 PM 3 query done: value
Waiting for 6 minutes...
4:50 PM 4 query done: value
Waiting for 6 minutes...
4:56 PM 5 query done: value
Waiting for 6 minutes...
5:02 PM 6 query done: value
Waiting for 6 minutes...
5:08 PM 7 query done: value
Waiting for 6 minutes...
5:14 PM 8 query done: value
Waiting for 6 minutes...
5:20 PM 9 query done: value
Waiting for 6 minutes...
5:26 PM 10 query done: value
Waiting for 6 minutes...

Going to have to stop it there because I need to stop all Apps and close or IDEs to clear my local NuGet package cache.