I won’t know without a repro, ideally a minimal stand alone repro that just uses ServiceStack.Redis library like you’re showing in your post so it clearly isolates where the issue is.
The project you’ve posted earlier is hidden with a lot of indirection and abstraction that makes it difficult to debug and harder to trace if the client is being misused. The purpose of the repro should be to show that there’s an issue with the ServiceStack.Redis library.
Again the same problem happens…This time it happened after I run the application for two days. The value of a key in one hashtable becomes the key of another hash table.
If data is corrupted you have a race condition where you still must be sharing the same connection between 2 threads somewhere.
I’ve added a feature that can be enabled to throw when it detects that the Redis client is being accessed from a different thread then the thread that resolved the client from the pool which can be enabled with:
RedisConfig.AssertAccessOnlyOnSameThread = true;
This captures the Thread’s StackTrace each time the client is resolved from the pool which is very slow, so should only be enabled when debugging connection issues.
If it does detect the client is being accessed from a different thread it will throw a InvalidAccessException with the message containing the different Thread Ids and the original StackTrace where the client was resolved from the pool. You can compare this with the StackTrace of the Exception to hopefully identify where the client is being improperly used.
This change is available in the latest v5.1.1 that’s now available on MyGet.
If you already have v5.1.1 installed you’ll need to clear your NuGet cache to download the latest version:
$ nuget locals all -clear
Does this new component require all these dependent dlls?
System.buffer.dll,
system.memory.dll,
system.numeric.vectors.dll
System.Runtime.CompilerServices.Unsafe.dll
My application is .net framework 4.5 built. Any dependency with 4.6?
ServiceStack.Text JSON and Text serializers were rewritten in v5.1.1 to use .NET’s new efficient Span<T> types which requires a dependency on System.Memory NuGet package which supports .NET v4.5.
ServiceStack.Redis still runs on .NET v4.5 and doesn’t have any dependencies on v4.6.
Thanks for the fix provided. I put it in our server and was watching for few days. I found few times invalidaccessexception in setvalue function. Though it is throwing exception, there is a retry logic written so it is working so far.
Is there any way to address without retrying logic?
Now another problem persists. when it is trying to read values, it returns either 0 or 1 (if the value does not exists) which troubles always. How to resolve this?
Any advice?.
Note I didn’t add a fix, I just added a debug option for trying to detect when the same Redis client instance is used across multiple threads that will include a StackTrace for where the client was resolved whilst the main StackTrace shows where another Thread is using it. The purpose is to help track down where the client is improperly used at which point you can disable it since capturing StackTraces is slow.
ServiceStack.Redis has built-in Automatic Retry, there are some configuration options to tweak the configuration.
This means you still have places where the redis client is improperly used where the client that uses the APIs is on a different Thread to where the client was originally resolved from the pool, refer to my comment on trying to track it down.
Which API returns 0 or 1? ServiceStack.Redis is just returning the value that’s returned from Redis, this behavior isn’t configurable. But if this is due to data corruption, you need to fix the places where the client is being improperly used.
have a doubt. Why the invalidaccessexception is not being caught in catch block?
Eg:-
function string test()
{
try
{
using (var client = m_pooledRedisManager.GetClient())
{
return client.GetValue(key);
}
}catch(Exception e)
{
console.write(“Exception”);
}
}
InvalidAccessException gets caught like any other normal Exception in C#, so it’s not being thrown within the try/catch handler also I’m assuming this is some pseudo code example as console.write will not compile.