Best Wire format for text heavy Poco Data

#1

Were interested in optimizing our wire serialization format for a returned List of POCO objects. Each POCO contains a string thats a key into a remote database that we cant change.

The key is ASCII and over 225 characters long and most often 90% of it repeats between POCO instances… So if we need to return 1000’s of them thats a lot of wasted data.

If we were able to use basic text compression on the entire JSON we could get a very high rate of compression.

Question: Are any of the existing wire formats, better than others at compressing data like this?

#2

All the binary formats, i.e. MsgPack, ProtoBuf and Wire wont repeat the key names.

CSV wont repeat key names either, but i’d limit its use for returning tabular datasets so should work well with List of POCO objects.

All other text serializers will repeat the key names.

#3

Sorry, i was not clear and i believe you misunderstood my question.

The value that I refer to as a key, is in contained in a string. So key Deduplication is not going to effect anything,

But i think i found an answer. In the book Mastering Service stack, they have a section on Compressing Responses.

It shows how to use a PreRequestFilter to Compress and Decompress a stream.

The client side is using a JsonServiceClient() which appears to has a property called DisableAutoCompression. So it sounds like if will detect
a compressed stream it would decompress it?

Also, are any of these advanced wire protocols supported by JsonServiceClient()? is there a different client to use in that case?

Thanks for all of your help!

#4

Yeah all the Service Clients transparently support deserializing compressed responses.

Also all the Server Caching and HTTP Caching both cache and return compressed responses.

But what code in PreRequestFilter are you using to compress responses?

#5

The normal way to cache responses in ASP.NET is to use a HttpModule.

However to make it easier to compress adhoc responses I’ve just added a new [CompressResponse] Response Filter attribute which you can use annotate a collection of Services or individual Services, e.g:

[CompressResponse]
public class MyCompressedServices : Service
{
}

public class MyServices : Service
{
    [CompressResponse]
    public object Any(MyCompress request) => ...;
}

This change is available from v4.5.7+ that’s now available on MyGet.

HTTP Response Compression with Self Hosted