What are the exact steps to repro? Running it just returns:
Still can’t repro it, but you absolutely should not be using the client after disposing it, e.g:
private bool SetEntryInHashExtn<T>(string hashid, string key, T value)
{
var bReturn = false;
IRedisTypedClient<T> redis;
using (var client = GetClient(true))
{
redis = client.As<T>();
}
var hash = redis.GetHash<string>(hashid);
bReturn = redis.SetEntryInHash(hash, key, value);
return bReturn;
}
All access needs to be done before disposing the client and returning it to the pool:
private bool SetEntryInHashExtn<T>(string hashid, string key, T value)
{
using (var client = GetClient(true))
{
var redis = client.As<T>();
var hash = redis.GetHash<string>(hashid);
return redis.SetEntryInHash(hash, key, value);
}
}
Steps to reproduce
- Change the redis sentinel connection string in RedisSentinelAdapter.cs file (please make sure connection go through sentinel only)
- Run the application from two machine at same time
- Let it run for few hours.
You can see the values are interchanging between two hashtables (Please refer untitled.png file for reference).
Please update all your code to only access the redis client instance within a using statement as per my example, this has the same behavior as maintaining a reference to a redis server collection beyond the using scope I described above which would cause this issue.
I am doing that now. I will update you in few minutes with the results.
Sometimes I am able to reproduce it within seconds also!!!
As per your suggestion, I have modified the code everywhere and tried in sample application. So far it is working fine. Earlier I used to get the problem in few minutes.
I will merge this changes with development branch so that I can test from more machines in different context, and I will update you the status.
Are you able to reproduce it?
Thank you very much for your feedback.
This is exactly the type of incorrect usage I’m referring to throughout this thread which causes this exact issue reported. AFAIC this is the cause, if you have a verified repro with correct usage of the library I’ll spend more time re-investigating.
Can you look at the function: SearchKeysExtn(string pattern)
I have same issue here also. It returns junk values few times. Is there any code change required here? if not, why this happens?
The issue isn’t localized to just the methods with incorrect usage, if there’s anywhere that uses the Redis instance after its returned to the pool, it could result in corrupted response the next time the same instance is resolved and used.
But I am referring the redis client instance within these (redissentineladapter.cs) functions only. And each time I am disposing it.
I have changed all functions as you suggested.
Now my problem is, some time when I read some values from Redis it gives values like “0” OR “1” (intermittent). What could be the reason?
The function is (The function is there in the redissentineladapter.cs file):
private T GetValueFromHashExtn<T>(string hashID,string key)
{
using(var client = m_redismanager.Getclient())
{
var redis = client.As<T>();
var hash = redis.GetHash<string>(hashid)
return hash[key];
}
}
I won’t know without a repro, ideally a minimal stand alone repro that just uses ServiceStack.Redis library like you’re showing in your post so it clearly isolates where the issue is.
The project you’ve posted earlier is hidden with a lot of indirection and abstraction that makes it difficult to debug and harder to trace if the client is being misused. The purpose of the repro should be to show that there’s an issue with the ServiceStack.Redis library.
Again the same problem happens…This time it happened after I run the application for two days. The value of a key in one hashtable becomes the key of another hash table.
If data is corrupted you have a race condition where you still must be sharing the same connection between 2 threads somewhere.
I’ve added a feature that can be enabled to throw when it detects that the Redis client is being accessed from a different thread then the thread that resolved the client from the pool which can be enabled with:
RedisConfig.AssertAccessOnlyOnSameThread = true;
This captures the Thread’s StackTrace each time the client is resolved from the pool which is very slow, so should only be enabled when debugging connection issues.
If it does detect the client is being accessed from a different thread it will throw a InvalidAccessException with the message containing the different Thread Ids and the original StackTrace where the client was resolved from the pool. You can compare this with the StackTrace of the Exception to hopefully identify where the client is being improperly used.
This change is available in the latest v5.1.1 that’s now available on MyGet.
If you already have v5.1.1 installed you’ll need to clear your NuGet cache to download the latest version:
$ nuget locals all -clear
Does this new component require all these dependent dlls?
System.buffer.dll,
system.memory.dll,
system.numeric.vectors.dll
System.Runtime.CompilerServices.Unsafe.dll
My application is .net framework 4.5 built. Any dependency with 4.6?
ServiceStack.Text JSON and Text serializers were rewritten in v5.1.1 to use .NET’s new efficient Span<T> types which requires a dependency on System.Memory NuGet package which supports .NET v4.5.
ServiceStack.Redis still runs on .NET v4.5 and doesn’t have any dependencies on v4.6.
Thanks for the fix provided. I put it in our server and was watching for few days. I found few times invalidaccessexception in setvalue function. Though it is throwing exception, there is a retry logic written so it is working so far.
Is there any way to address without retrying logic?
Now another problem persists. when it is trying to read values, it returns either 0 or 1 (if the value does not exists) which troubles always. How to resolve this?
Any advice?.
Note I didn’t add a fix, I just added a debug option for trying to detect when the same Redis client instance is used across multiple threads that will include a StackTrace for where the client was resolved whilst the main StackTrace shows where another Thread is using it. The purpose is to help track down where the client is improperly used at which point you can disable it since capturing StackTraces is slow.
ServiceStack.Redis has built-in Automatic Retry, there are some configuration options to tweak the configuration.
This means you still have places where the redis client is improperly used where the client that uses the APIs is on a different Thread to where the client was originally resolved from the pool, refer to my comment on trying to track it down.
Which API returns 0 or 1? ServiceStack.Redis is just returning the value that’s returned from Redis, this behavior isn’t configurable. But if this is due to data corruption, you need to fix the places where the client is being improperly used.
have a doubt. Why the invalidaccessexception is not being caught in catch block?
Eg:-
function string test()
{
try
{
using (var client = m_pooledRedisManager.GetClient())
{
return client.GetValue(key);
}
}catch(Exception e)
{
console.write(“Exception”);
}
}
InvalidAccessException gets caught like any other normal Exception in C#, so it’s not being thrown within the try/catch handler also I’m assuming this is some pseudo code example as console.write will not compile.