Redis data returned is not what was expected

#1

Using this code:

private string GetKeyForId(int idShipmentRequest)
{
  return $"{key}:{idShipmentRequest}";
}

public DTO.AnnouncedShipment GetById(int id)
{
    using (var client = manager.GetClient())
    {
      var announcedShipments = client.As<DTO.AnnouncedShipment>();
      var key = GetKeyForId(id);
      var announcedShipment = announcedShipments.GetValue(key);
      logger.Info($"GetById for id: {id}. Key: {key}. Announced: {announcedShipment.Dump()}");
      return announcedShipment;
    }
}

we sometime see this in our log:

2019-11-05 12:24:46.0095||ShipmentLogger|INFO|GetById for id: 14443493. Key: AnnouncedShipment:14443493. Announced: {
	idShipmentRequest: 14443493,
	idContact: 65178992,
	idCampagne: 1921,
	packingDetails: 
	[
		{
			idPackingType: 3,
			numberOfItems: 2
			....

This is correct, since the so called idSHipmentRequest should be identical to the id we get the value for.
But sometimes we see this - the same request, different result:

2019-11-05 14:11:14.1342||ShipmentLogger|INFO|GetById for id: 14443493. Key: AnnouncedShipment:14443493. Announced: {
	idShipmentRequest: 14460000,
	idContact: 64795584,
	idCampagne: 1088,
	packingDetails: 
	[
		{
			idPackingType: 4,
			numberOfItems: 2
			...

What can cause this? Is this a REDIS issue? Of could it be ServiceStack?

#2

Couldn’t tell you, it’s just reporting what’s stored in Redis, have you verified against what’s stored in Redis?

#3

Yes, I verified and double checked. In fact, we have been hunting this ‘feature’ for a couple of months and it happens every now and then. We are now actively monitoring REDIS as well to see if the data returned is actually the wrong value, or if the issue lies with the RedisClient.
Either way, it seems to be a buffering thing. Reason I suspect buffering is that the data returned in a consecutive call to this or another pooled client yields the data for the past request. So basically, if we ask data for 1, 2, 3, 4, 5 we sometimes get 2, 3, 4, 5.
Still looking into this, but it’s not something you have seen before?

#4

No, I’ve no idea of what the underlying cause could be.

#5

We ran into a similar issue and based on our findings we got a successful return from Redis for a given key but the data returned was from a different key. I was curious about Rob’s setup. We are running the Windows version of Redis (for now) on a 3 node cluster. At the time of this event the primary was failing over to one of the backup nodes and all occurrences of this event happened within a 3 minute window of the failover event. If @Rob is on a Linux instance it may help determine where the issue lies?

#6

We run both Linux and Windows version.
Windows for DEV and Linux for PROD. Our findings were with the Linux version.
We ‘fixed’ this by requesting the value after we know it’s not the right value. This seems to fix the issue.

#7

Thanks for the info Rob! We are implementing a similar strategy. I feel like the fact that we are on different platforms points to an issue in the component.

#8

@Rob one more question. Would you mind sharing what version of ServiceStack you are on? Thanks!

#9

5.7.0 but we saw the same behaviour with 5.5.0. I suspect this is a REDIS ‘feature’ though