Hello, I have 3 Redis Sentinel instances and when I initialize like below the ReadPoolIndex and WritePoolIndex of the PooledRedisClientManager are still 0 which doesn’t allow it to work.
This is also my first time using Sentinels, can’t find it as a configuration option.
How do you manage these indexes?
Thanks
var sentinel = new RedisSentinel(AppSettings.GetList(RedisDevSentinel), "mymaster");
var redisManager = sentinel.Start();
container.Register<IRedisClientsManager>(redisManager);
If AppSettings.GetList(RedisDevSentinel) returns the host:port of your Redis Sentinel Instances then that should be all that’s needed to start resolving Redis Clients for the configured redisManager, e.g:
using var redis = redisManager.GetClient();
What’s the actual Issue/Exception?
This is the first issue, I believe this is from ServiceStack.RateLimit.Redis
“errorCode”: “InvalidOperationException”,
“message”: “Need a minimum read pool size of 1, then call Start()”,
“stackTrace”: "System.InvalidOperationException: Need a minimum read pool size of 1, then call Start()\r\n at ServiceStack.Redis.PooledRedisClientManager.AssertValidReadOnlyPool() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\PooledRedisClientManager.cs:line 716\r\n at ServiceStack.Redis.PooledRedisClientManager.
Where are you seeing ServiceStack.RateLimit.Redis from? I don’t know of any ServiceStack.Redis type with that namespace.
Are you using the standard Redis Sentinel Setup of 3x Sentinels, 2x Slaves, 1x Master? Which is also our recommended setup in the ServiceStack/redis-config repo.
Can you check that all your Redis instances are configured and running correctly. Then check that Redis Sentinel is returning the correct master instance by connecting to a Sentinel Instance and asking for the information it has about the master instance, e.g:
$ redis-cli -p 5000
$ sentinel master mymaster
Yes we have 3x Sentinels, 2x Slaves, 1x Master, we are using K8s and not VMs
$ redis-cli -p 26379
127.0.0.1:26379> sentinel master mymaster
1) "name"
2) "mymaster"
3) "ip"
4) "[NAME REMOVED]"
5) "port"
6) "6379"
7) "runid"
8) "ca619dd4ea23fe0e243ece5846bec0907150afc7"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "952"
19) "last-ping-reply"
20) "952"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "1312"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "60190651"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "30000"
39) "parallel-syncs"
40) "1"
Can you try directly connecting to this master instance from the App, e.g:
var redisManager = new RedisManagerPool("<ip>:6379");
using var redis = redisManager.GetClient();
redis.Info.PrintDump();
If you can connect can you try enable debug logging to see if that shows anything:
LogManager.LogFactory = new ConsoleLogFactory(debugEnabled:true);
Connecting directly to the master looks good.
DEBUG: New Redis Masters: [name removed]:6379
DEBUG: New Redis Replicas:
DEBUG: #2 Attempting sync connection to ‘[name removed]:6379’ (SEND -1, RECV -1 Timeouts)…
DEBUG: #2 Socket connected to ‘[name removed]:6379’
{
redis_version: 6.2.6,
redis_git_sha1: 00000000,
redis_git_dirty: 0,
redis_build_id: 5b326c4b337eb561,
redis_mode: standalone,
os: Linux 4.18.0-348.23.1.el8_5.x86_64 x86_64,
arch_bits: 64,
multiplexing_api: epoll,
atomicvar_api: c11-builtin,
gcc_version: 8.3.0,
process_id: 9,
process_supervised: no,
run_id: 99543e98236f3cc602a8053ae750e61af283d3c6,
tcp_port: 6379,
server_time_usec: 1686927513285205,
uptime_in_seconds: 66191,
uptime_in_days: 0,
hz: 10,
configured_hz: 10,
lru_clock: 9205913,
executable: /redis-server,
config_file: /var/tmp/redis.conf,
io_threads_active: 0,
connected_clients: 7,
cluster_connections: 0,
maxclients: 10000,
client_recent_max_input_buffer: 32,
client_recent_max_output_buffer: 0,
blocked_clients: 0,
tracking_clients: 0,
clients_in_timeout_table: 0,
used_memory: 2091160,
used_memory_human: 1.99M,
used_memory_rss: 15032320,
used_memory_rss_human: 14.34M,
used_memory_peak: 2253184,
used_memory_peak_human: 2.15M,
used_memory_peak_perc: 92.81%,
used_memory_overhead: 2043216,
used_memory_startup: 810016,
used_memory_dataset: 47944,
used_memory_dataset_perc: 3.74%,
allocator_allocated: 2183544,
allocator_active: 2551808,
allocator_resident: 10252288,
total_system_memory: 809901527040,
total_system_memory_human: 754.28G,
used_memory_lua: 37888,
used_memory_lua_human: 37.00K,
used_memory_scripts: 0,
used_memory_scripts_human: 0B,
number_of_cached_scripts: 0,
maxmemory: 1000000000,
maxmemory_human: 953.67M,
maxmemory_policy: allkeys-lfu,
allocator_frag_ratio: 1.17,
allocator_frag_bytes: 368264,
allocator_rss_ratio: 4.02,
allocator_rss_bytes: 7700480,
rss_overhead_ratio: 1.47,
rss_overhead_bytes: 4780032,
mem_fragmentation_ratio: 7.06,
mem_fragmentation_bytes: 12902024,
mem_not_counted_for_evict: 0,
mem_replication_backlog: 1048576,
mem_clients_slaves: 41024,
mem_clients_normal: 143528,
mem_aof_buffer: 0,
mem_allocator: jemalloc-5.1.0,
active_defrag_running: 0,
lazyfree_pending_objects: 0,
lazyfreed_objects: 0,
loading: 0,
current_cow_size: 0,
current_cow_size_age: 0,
current_fork_perc: 0.00,
current_save_keys_processed: 0,
current_save_keys_total: 0,
rdb_changes_since_last_save: 2,
rdb_bgsave_in_progress: 0,
rdb_last_save_time: 1686861322,
rdb_last_bgsave_status: ok,
rdb_last_bgsave_time_sec: 0,
rdb_current_bgsave_time_sec: -1,
rdb_last_cow_size: 389120,
aof_enabled: 0,
aof_rewrite_in_progress: 0,
aof_rewrite_scheduled: 0,
aof_last_rewrite_time_sec: -1,
aof_current_rewrite_time_sec: -1,
aof_last_bgrewrite_status: ok,
aof_last_write_status: ok,
aof_last_cow_size: 0,
module_fork_in_progress: 0,
module_fork_last_cow_size: 0,
total_connections_received: 13263,
total_commands_processed: 444207,
instantaneous_ops_per_sec: 8,
total_net_input_bytes: 30489761,
total_net_output_bytes: 197900982,
instantaneous_input_kbps: 0.56,
instantaneous_output_kbps: 4.77,
rejected_connections: 0,
sync_full: 2,
sync_partial_ok: 0,
sync_partial_err: 2,
expired_keys: 0,
expired_stale_perc: 0.00,
expired_time_cap_reached_count: 0,
expire_cycle_cpu_milliseconds: 735,
evicted_keys: 0,
keyspace_hits: 2,
keyspace_misses: 1,
pubsub_channels: 1,
pubsub_patterns: 0,
latest_fork_usec: 386,
total_forks: 2,
migrate_cached_sockets: 0,
slave_expires_tracked_keys: 0,
active_defrag_hits: 0,
active_defrag_misses: 0,
active_defrag_key_hits: 0,
active_defrag_key_misses: 0,
tracking_total_keys: 0,
tracking_total_items: 0,
tracking_total_prefixes: 0,
unexpected_error_replies: 0,
total_error_replies: 0,
dump_payload_sanitizations: 0,
total_reads_processed: 448237,
total_writes_processed: 820280,
io_threaded_reads_processed: 0,
io_threaded_writes_processed: 0,
role: master,
connected_slaves: 2,
slave0: “ip=[name removed],port=6379,state=online,offset=21147225,lag=1”,
slave1: “ip=[name removed],port=6379,state=online,offset=21147225,lag=1”,
master_failover_state: no-failover,
master_replid: 90ad50ede72a67f95d139b2514901767fb12c09d,
master_replid2: d288d7ca35f09be803868bd39f9c2bff92dcb8e9,
master_repl_offset: 21147447,
second_repl_offset: 1553183,
repl_backlog_active: 1,
repl_backlog_size: 1048576,
repl_backlog_first_byte_offset: 20098872,
repl_backlog_histlen: 1048576,
used_cpu_sys: 32.777913,
used_cpu_user: 50.906667,
used_cpu_sys_children: 0.001801,
used_cpu_user_children: 0.001149,
used_cpu_sys_main_thread: 32.505741,
used_cpu_user_main_thread: 50.570839,
cluster_enabled: 0,
db0: “keys=1,expires=0,avg_ttl=0”
}C:\dev\RedisDebugging\RedisDebugging\bin\Debug\net6.0\RedisDebugging.exe (process 10368) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .
What’s the debug logging when connecting with RedisSentinel?
It so happened to be an issue with the configuration of the Sentinels, all set now on that front, thanks and sorry for crying wolf.
Though I can’t see how to IoC register the IRedisClientsManagerAsync
container.Register<IRedisClientsManagerAsync>( ? )
The PooledRedisClientManager implements the Async interface as well, by default this is what is used by RedisSentinel.
Thanks for the reply, if I need to use IRedisClientsManagerAsync via IoC from another class how would I access it?
If you are using it from RedisSentinel, you can do something like
var redisManager = sentinel.Start();
var client = await ((IRedisClientsManagerAsync)redisManager).GetClientAsync();
Assuming you are using a clients manager that supports both interfaces like the default PooledRedisClientManager used by RedisSentinel by default.
If you only need the async client for some operations you can also access the async client using the extension method GetClientAsync which is probably easier:
var clientAsync = await redisManager.GetClientAsync()
Perfect, thank you very much!