Redis Byte Array Operations

1

One of the things I need to do is to get/set/subscribe to deal with byte arrays. So far, I have:

    ServiceStack.Redis.PooledRedisClientManager.DisposablePooledClient<ServiceStack.Redis.RedisNativeClient> redisNativeClient;
        redisNativeClient = redisFactory.GetDisposableClient<ServiceStack.Redis.RedisNativeClient>();

…

        byte[] baResponse = redisNativeClient.Client.Get(sChannel);

…

    public void PublishAndSetValue(string sChannel, byte[] baValues)
    {
        redisNativeClient.Client.Publish(sChannel, baValues);
        redisNativeClient.Client.Set(sChannel, baValues);
    }

…

And this is the part where I’m stuck:

        redisFactory = new ServiceStack.Redis.PooledRedisClientManager(sHost);
        redisPubSubServer = new ServiceStack.Redis.RedisPubSubServer(redisFactory, saSubscribeToChannels);
        redisPubSubServer.OnMessage += RedisMessageReceivedHandler;
        redisPubSubServer.Start();

OnMessage is Action<string, string>.

Is there Action<string, byte[]> OnMessageBytes or anything like that? I’d hate to convert the string back to a byte array for performance reasons.

2

No there’s only a string so you’d need to either send the payload as base64 or do something like send a key that references a byte value that subscribers download themselves as would be the optimal strategy for sending large payloads.

3

They’re not large, rather there’s many, many small ones. The back and forth from byte array to UTF8 creates problems in that it’s not reversible, for example - the following breaks on round trip

I would be inclined to call this a bug.

    void DoStuff()
    {
        byte[] baEncoded = GetBytesForData(); //40 Bytes
        string sToString = Encoding.UTF8.GetString(baEncoded);
        byte[] baDecoded = Encoding.UTF8.GetBytes(sToString); //68 Bytes!!!!
    }
    

    protected unsafe byte[] GetBytesForData()
    {
        int iByteSize = sizeof(long);
        if (iByteSize != sizeof(double)) throw new DataMisalignedException();
        byte[] array = new byte[5 * iByteSize];
        fixed (byte* byteptr = array)
        {
            double* dblptr = (double*)byteptr;
            long* longptr = (long*)byteptr;
            longptr[0] = 1;
            dblptr[1] = 2.3;
            dblptr[2] = 4.5;
            dblptr[3] = 6.7;
            longptr[4] = -8;
        }
        return array;
    }
4

You should be sending binary over text as a Base64 string, i.e. Use Convert.ToBase64String() and Convert.FromBase64String()

5

I thought strings in Redis were binary-safe, no?

6

This has nothing to do with Redis.

You can’t use UTF-8 text encoding to encode binary data, it’s for encoding text which C# strings stores as UTF-16 surrogate pairs into UTF-8 bytes.

By far the most popular way to encode binary data to text in .NET is to use Base64, which is what you should be using instead.

7

Unfortunately, I’m not the sole user of this Redis database, and my role is rather limited. The other people using it are not even using .NET, they’re using C on Linux. Their idea is to pull the bytes, and do something like this:

struct SOMEData
{
__int64 someInt;
double someDoubles[3];
__int64 otherInt;
};

void UnpackBytes(unsigned char* data)
{
SOMEDatasomeData = (SOMEData)data;
…
}

Assuming the other users won’t go along with the Base64 paradigm, that’s why I was wondering if there should be Action<string, byte[]> OnMessageBytes or if I’ll need to handle RedisNativeClient to get the bytes.

8

You wont be able to transfer arbitrary binary data over strings without encoding it, i.e. you can’t unpack bits from a string.

But I’ve added OnMessageBytes to both RedisSubscription and RedisPubSubServer in this commit which will let you read the message body as bytes.

This change is available from v5.5.1 that’s now available on MyGet.

9

Yes, excellent point. I hate that Redis calls the data type a string when it can be used to transfer non-string bytes. But anyway, this change is working beautifully well, much appreciated!

1 Like