Hi All @layoric and @mythz
After investigating, we found that the issue is related pool.
We have an IdentityServer service that many users access to store and retrieve data from Redis continuously. At the same time, another service is sequentially retrieving data from Redis, which causes the error.
Log from IdentityServer
Log from other Service
Detail for the error
2023-07-18 16:44:37.3290 [ERROR] [09c07b3f-fc5e-4d36-8879-438114eed308] Xtremax.Cache.RedisCacheClientServiceStack Failed to retrieve cache with key '09c07b3f-fc5e-4d36-8879-438114eed308/84d6d4ee-494e-4070-a2f3-a556bd937f75', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
2023-07-18 16:44:35.2300 [ERROR] [09c07b3f-fc5e-4d36-8879-438114eed308] Xtremax.Cache.RedisCacheClientServiceStack Failed to retrieve cache with key '09c07b3f-fc5e-4d36-8879-438114eed308/84d6d4ee-494e-4070-a2f3-a556bd937f75', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
To replicate this, I created a sample service called SampleRedis, which performs concurrent store and retrieve operations. Another service hits SampleRedis using HttpClient. I used 3 sample services and 3 Replicas Redis Server (Master/Slave) and 3 Redis Sentinel.
SampleRedis Code
Sample Service when Hit SampleRedis Code
I tried hitting Service 1 100 times, Service 2 50 times, and Service 3 30 times.
During testing, the error occurred and it only happened on Service 3
2023-07-25 04:35:41.7051 [ERROR] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Failed to store cache with key 'AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use."}
at ServiceStack.Redis.PooledRedisClientManager.GetClient() in C:\\BuildAgent\\work\\b2a0bfe2b1c9a118\\src\\ServiceStack.Redis\\PooledRedisClientManager.cs:line 262"}
Xtremax.Cache.RedisCacheClientServiceStack._Store(String key, Object value)"}
Xtremax.Cache.RedisCacheClientServiceStack.Store(String key, Object value)"}
2023-07-25 04:35:41.7051 [INFO] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Start storing cache with key : AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1
2023-07-25 04:35:42.7057 [ERROR] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Failed to store cache with key 'AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use."}
at ServiceStack.Redis.PooledRedisClientManager.GetClient() in C:\\BuildAgent\\work\\b2a0bfe2b1c9a118\\src\\ServiceStack.Redis\\PooledRedisClientManager.cs:line 262"}
Xtremax.Cache.RedisCacheClientServiceStack._Store(String key, Object value)"}
Xtremax.Cache.RedisCacheClientServiceStack.Store(String key, Object value)"}
2023-07-25 04:35:42.7057 [INFO] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Start storing cache with key : AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1
2023-07-25 04:35:42.7057 [INFO] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Start storing cache with key : AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1
2023-07-25 04:35:43.7054 [ERROR] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Failed to store cache with key 'AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use."}
at ServiceStack.Redis.PooledRedisClientManager.GetClient() in C:\\BuildAgent\\work\\b2a0bfe2b1c9a118\\src\\ServiceStack.Redis\\PooledRedisClientManager.cs:line 262"}
Xtremax.Cache.RedisCacheClientServiceStack._Store(String key, Object value)"}
Xtremax.Cache.RedisCacheClientServiceStack.Store(String key, Object value)"}
2023-07-25 04:35:43.7054 [INFO] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Start storing cache with key : AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1
2023-07-25 04:35:44.7060 [ERROR] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Failed to store cache with key 'AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use."}
at ServiceStack.Redis.PooledRedisClientManager.GetClient() in C:\\BuildAgent\\work\\b2a0bfe2b1c9a118\\src\\ServiceStack.Redis\\PooledRedisClientManager.cs:line 262"}
Xtremax.Cache.RedisCacheClientServiceStack._Store(String key, Object value)"}
Xtremax.Cache.RedisCacheClientServiceStack.Store(String key, Object value)"}
2023-07-25 04:35:44.7060 [INFO] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Start storing cache with key : AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1
2023-07-25 04:35:44.7060 [ERROR] [77fa673f-513a-4723-853f-14b8e9bca70a] Xtremax.Cache.RedisCacheClientServiceStack Failed to store cache with key 'AuthSessionStore-f55f070a-713c-4ab0-a515-a3c338f8a5c1', reason : Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use. System.TimeoutException: Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use."}
at ServiceStack.Redis.PooledRedisClientManager.GetClient() in C:\\BuildAgent\\work\\b2a0bfe2b1c9a118\\src\\ServiceStack.Redis\\PooledRedisClientManager.cs:line 262"}
Xtremax.Cache.RedisCacheClientServiceStack._Store(String key, Object value)"}
Xtremax.Cache.RedisCacheClientServiceStack.Store(String key, Object value)"}
We have followed the documentation and followed the instructions, such as using singleton and using statements.
Our code for using Redis Client like these
Singletone
Store Process
Retrieve Process
Based on my stimulate or replicate the issue and the code above, do you have any suggestions or improvements?
(Expected)
(This means I deleted all Redis pod servers)
(Not Expected)