I am facing an issue getting junk values like ‘OK’ or ‘0’ or some ‘numeric’ values while reading values from Redis. This happens while reading normal key and hash keys as well. We have upgraded all the service stack components and still facing the issues. Please find the component details and the code snippet in c#
Our environment: We have setup sentinel, and each sentinel is paired with a Redis instance. Now we have setup three instances of sentinel and three instances of redis server. We were using read only client for reading values and read-write client for writing values to Redis. Even after using read-write client for reading and writing is also giving the same junk value problems.
Components:
ServiceStack.Common v 5.1.0.0
ServiceStack.Redis v 5.1.0.0
ServiceStack.Interfaces v 5.1.0.0
ServiceStack.Text v 5.1.0.0
Redis server v 3.0.503
OS: Windows server 2012 R2
code snippet (RedisManager class):
private static IredisClientManager m_redisManager;
initializeRedis()
{
if(m_redisManager == null)
{
var sentinel = new RedisSentinel(
"193.168.1.1:16380,193.168.1.2:16380,193.168.1.3:16380"
,"testmaster")
{
RefreshSentinelHostsAfter = 10;
};
sentinel.RedisManagerFactory += (master,slaves)
=> new RedisManagerPool(master);
m_redisManager = sentinel.Start();
}
}
public T GetValueFromHash<T>(string hashid, string key)
{
var result = default(T);
using(var client = m_redisManager.GetClient())
{
var redis = client.As<T>();
var hash = redis.GetHash<string>(hashid);
result = hash[key];
}
return result;
}
client:
RedisManager manager = new RedisManager ();
var val = manager.GetValueFromHash(“devicehash”,“devicekey”);
Note:
m_redisManager is declared as static, so that it runs only once. Each call will share this manager 2. client is disposing after each call to get value 3. My application is a multi threaded, so reading from multiple thread may happen at the same time. And application is muti instancing from same machine and difeerent machine as well. 4. The above code is from component which interact with Redis. 5. Client will call GetValue function
Getting junk/corrupted values is a typically an indication you’re using the same non-ThreadSafe RedisClient instance across multiple threads. This is prevented if you only used using(){...} statements in all your different threads, e.g. like you’re doing here:
using(var client = m_redisManager.GetClient())
{
var redis = client.As<T>();
var hash = redis.GetHash<string>(hashid);
result = hash[key];
}
But since you’re not in this example, it wont be reproducible with what’s provided.
Unfortunately when using the same RedisClient instance across multiple threads you can’t easily determine the cause where the RedisClient was used incorrectly. But basically you need to ensure you’re always using the RedisClient in a using(){...} statement and that you’re never maintaining a RedisClient instance in a static field or using the same instance in a background thread. Each thread should resolve its own instance from the redisManager and dispose of it when its done.
If you can put together a stand-alone repro I can run locally to reproduce the issue (e.g. in a GitHub project or Gist) I will be able to determine what the issue is.
I am not caching any redisclient objects. As I mentioned in the previous post, I am caching only IredisClientManager object (see the code snippet: private static IredisClientManager m_redisManager;). As you can see in the code snippet, I have used using(){…} statement while creating client object.
I strictly followed all guidelines you have suggested and still it is happening. The most frustrating thing is it is not happening always.
So please look into this issue seriously and give us a solution ASAP.
In all cases this has been reported it’s always been misuse of the client instance. My suggestion is common things to look out for, I don’t have access to your source code or your environment to know what’s actually causing your issue.
Another thing to look out for was if you’re saving any redis server collections, e.g:
var list = redis.Lists["mylist"];
As the collection wraps a redis client so when you dispose the client it returns it to the pool, so the next time the list is accessed it could also be concurrently accessed the next time the client is retrieved from the pool with redisManager.GetClient(). The solution in this case if you need access to the List contents you will need to retrieve a local disconnected copy of the list with:
var list = redis.Lists["mylist"].GetAll();
We absolutely need a stand-alone verifiable repro we can run locally in order to be able to identify any issue. The issue doesn’t have to occur all the time, but it has to occur at least one time when running the repro for anyone to stand any chance at being able to identify it.
Since you’re using RedisSentinel one thing you can try is upgrading to v5.1.1 on MyGet which contains a fix for a locking issue on Sentinel failover. It didn’t exhibit this symptom so will unlikely have any affect, but it’s the easiest option you can try.
Note: Here I am creating two hash tables 1. statusHash 2. LocationHash.
While running application, I can see the value getting copied from locationHash to statusHash and vice versa (please refer the image). And sometimes the key returns “\0” and “\1” as well.
Please update all your code to only access the redis client instance within a using statement as per my example, this has the same behavior as maintaining a reference to a redis server collection beyond the using scope I described above which would cause this issue.
As per your suggestion, I have modified the code everywhere and tried in sample application. So far it is working fine. Earlier I used to get the problem in few minutes.
I will merge this changes with development branch so that I can test from more machines in different context, and I will update you the status.
Are you able to reproduce it?
Thank you very much for your feedback.
This is exactly the type of incorrect usage I’m referring to throughout this thread which causes this exact issue reported. AFAIC this is the cause, if you have a verified repro with correct usage of the library I’ll spend more time re-investigating.
The issue isn’t localized to just the methods with incorrect usage, if there’s anywhere that uses the Redis instance after its returned to the pool, it could result in corrupted response the next time the same instance is resolved and used.
But I am referring the redis client instance within these (redissentineladapter.cs) functions only. And each time I am disposing it.
I have changed all functions as you suggested.
Now my problem is, some time when I read some values from Redis it gives values like “0” OR “1” (intermittent). What could be the reason?
The function is (The function is there in the redissentineladapter.cs file):
private T GetValueFromHashExtn<T>(string hashID,string key)
{
using(var client = m_redismanager.Getclient())
{
var redis = client.As<T>();
var hash = redis.GetHash<string>(hashid)
return hash[key];
}
}
