Design - Handling Encrypted Messages

1

A design/recommended-practice question.

I am having to send confidential messages over HTTP, and will be using asymmetric encryption to ensure confidentiality of messages. The encrypted requests will contain various tokens etc to ensure the authenticity of the user, to be verified by the service implementation. Stand web authZ stuff.

The question is now, how best to do that with ServiceStack? i.e. what implementation pieces to use to do this elegantly?

I can see a couple implementation options:

  1. Define a Request and Response DTOs that includes a single unstructured text field containing the blob of encrypted data. Handle the decryption of that blob, stuff the unencrypted data into structured DTOS, and validate them all in the service implementation itself.(as sort of described in http://stackoverflow.com/a/16110300/2496582)
  2. Have a RequestFilter handle the encrypted blob request, and passing through the structured unencrypted DTO’s to other filters and the service implementation. Ideally the validation filter will fire next before the service implementation gets the request - like normal?

I like #2 above over #1 for many reasons, including:

  • We can easily apply this filter to only methods we need to.
  • We can create proper request DTOS with real structure of the data - as per usual.
  • We can utilize standard validation mechanisms as usual.
  • for all intents and purposes everything else is as usual, with just an additional decryption filter.

Is #2 even possible? any examples or guidance for how filters can handle data in one format (encrypted blob) and pass through new DTO’s in a different format, to other filters and then to the service implmentation?

2

I guess, essentially all I really need to know is this:

If I create a RequestFilter (to handle my encrypted DTO), can it pass through a different structured DTO (containing un-encrypted data) to subsequent filters?

3

You can’t change the Request DTO midway through executing request filters and then get it to travel back to the start of the request filters again with a different DTO. The place this needs to be done is when the Request DTO is created which you can do by overriding by adding a custom Request Binder.

I’ve had a go at implementing a close example of #2 in a stand-alone test in this commit.

Basically it uses a custom request binder to convert the EncryptedMessage Request DTO into the decrypted Request DTO which is then what’s used to execute down through all the request filters. After the Request DTO is converted we need to regain control of the response so we can convert it into an Encrypted Response Message that’s sent back to the client. Once on the client we can then just decrypt the returned encrypted response DTO to access the response.

Eventually I’d like to add this feature as a plugin so it can easily be re-used. One thing I need to add is a corresponding ResponseBinder so it behaves similar to the RequestBinder in that it can convert the Response DTO into a different DTO (e.g. EncryptedMessageResponse), i.e. so it doesn’t need to override an AppHost method which plugins can’t do.

I’d also need to come up with a strategy for managing the public/private keys and how the encryption is done and what properties need to be available on the EncryptedMessage Request DTO, so I’d like to know more about your use-case, i.e. are you planning on having a unique Key/Pair for each user or just a single Global KeyPair configured for the Server? What type of encryption do you plan to use and do you intent on using the main KeyPair to also encrypt the actual messages or use it to negotiate a new/faster encryption algo be used for the actual encryption of the messages?

4

This is quiet similar to what the SO answer uses. After decrypting the message and working out structured requests type info successfully, you could fire ApplyRequestFilters on the AppHost. I believe this will go through the same filtering process a normal request will go through.

Note: Only using Base64 here as an example of dealing with encoded messages, not at all suggesting this is in any way secure!

I tried this with a simple example and successfully fired filters associated with the structured decrypted request.

this.RegisterTypedRequestFilter<Hello>((request, response, arg3) =>
{
    if (arg3.Name == "Foo")
    {
        throw HttpError.Forbidden("No Foo!");
    }
});

this.RegisterTypedRequestFilter<Base64>((request, response, arg3) =>
{
    byte[] data = Convert.FromBase64String(arg3.Content);
    string decodedString = Encoding.UTF8.GetString(data);
    //Resolve type info...
    Hello newReq = JsonSerializer.DeserializeFromString<Hello>(decodedString);
    if (AppHost.Instance.ApplyRequestFilters(request, response, newReq))
    {
        throw HttpError.Forbidden("?");
    }
});

Another option (if hosting the unsecured endpoint separately is an option) might be to use custom serializer/deserializer on specific requests. This may not be practical for a number of reasons, but food for thought.

public override void Configure(Container container)
{
    //Config examples
    //this.Plugins.Add(new PostmanFeature());
    //this.Plugins.Add(new CorsFeature());

    JsConfig<Hello>.DeSerializeFn = s =>
    {
        byte[] data = Convert.FromBase64String(s);
        string decodedString = Encoding.UTF8.GetString(data);
        return JsonSerializer.DeserializeFromString<Hello>(decodedString);
    };

    JsConfig<HelloResponse>.SerializeFn = response =>
    {
        var responseString = JsonSerializer.SerializeToString(response);
        var result = Convert.ToBase64String(Encoding.UTF8.GetBytes(responseString));
        return result;
    };

    this.RegisterTypedRequestFilter<Hello>((request, response, arg3) =>
    {
        if (arg3.Name == "Foo")
        {
            throw HttpError.Forbidden("No Foo!");
        }
    });
}

For example, if I POST the following eyAiTmFtZSI6ICJGb28xIiB9 as the raw body with Content-Type as application/json the custom JsConfig<T>.DeSerialize method is fired before typed request filter and the rest of the usual SS workflow works with unencrypted DTOs (this may not be sufficient depending on your security requirements).

SerializeFn then works on the response object and returns encrypted response.

Depending on what information is payloads is sensitive (content, headers, etc) these methods may not be sufficient as handling exceptions unencrypted may be giving away info you don’t want to. Work arounds above are pretty clunky (and probably won’t suit your requirements), I think message level encryption functionality would be a great addition to the framework, but would take quite a lot of consideration to come up with a good approach that works with existing components like the JsonServiceClient and other functionality.

5

Thanks Demis, I am going to expore that sample further.

In terms of my needs and design:
I am constrained in that my client can only talk HTTP, but can call my service with JSON. It also runs the NetMF framework, so has limited memory and storage and processing power (battery juice).

We already have a HTTPS + oAuth service, but the client cannot access it, so I plan to stand up another service in front of that to use the HTTP channel, but use the same SSL certificate for encrypting the request from this client as used in the HTTPS+oAuth service. The HTTP service then forwards the verified payload to the HTTPS + oAuth service.

It works like this:

  1. The client downloads the SSL cert (public key only) from known URL of the HTTP service. no reason to protect that.
  2. The client constructs a DTO which contains two parts: The first part is the message itself, containing whatever fields we need in the message. The second part is for giving us authenticity of the client. It will include a session token (that was exchanged prior) a client identifier (which the client and service knows) , and a nonce used to prevent replays (and we also have a correlationId to link to a message exchange that this call is the last part of - in this case).
  3. The client puts all those secrets into a request DTO in plain text, along with the message payload, and then encrypts the whole lot with the public key of the SSL cert.
  4. client sends the request to HTTP service route
  5. Service receives a very bare DTO with just encrypted blob in it. Nothing else.
  6. service decrypts blob (with private key of SSL cert), then validates DTO for datatype compliance, checks the nonce, then processes secrets, and finally processes the payload if all good.
  7. If we are all good at this point, The HTTP service just forwards the payload part of the DTO to the HTTPS+oAuth service to do the real processing.

In our case, at this point, only a 200 response is given back to client, so no encryption required on the response. But if we did, then the client would need a cert as well, which could in theory be included in the request in the clear.

So we might envision a request DTO something like this:

class DoSomething : IReturn
{
/// encrypted message
public string Data{get;set;}

/// public key for response (if any)
public string ResponsePublicKey {get;set;}

}

Might be all it needs. I suppose you can include a string of the type of the un-encrypted DTO in there too, but I think the service ought to have that knowledge for the specific call.

Does that give you a sense of how it could hang together (for us at least) as a data point.?

6

Thanks @layoric I’ll consider that pattern after I give @mythz a go first.

cheers

7

Hi @jezzsantos as I wasn’t satisfied with the current functionality I used to implement this since the Request Binder is generally meant for overriding the binding and AppHost.OnAfterExecute requires subclassing the AppHost which plugins can’t do, it’s also not symmetrical with the Request Binder which ends up looking hacky so I’ve added a new support for Request and Response Converters which allows changing the return type used for Requests and Responses.

This is what the EncryptedMessages example looks like after changing to use it:

RequestConverters.Add((req, requestDto) => {
    var encRequest = requestDto as EncryptedMessage;
    if (encRequest == null)
        return null;

    var requestType = Metadata.GetOperationType(encRequest.OperationName);
    var decryptedJson = CryptUtils.Decrypt(SecureConfig.PrivateKey, encRequest.EncryptedBody);
    var request = JsonSerializer.DeserializeFromString(decryptedJson, requestType);
    req.Items["_encrypt"] = encRequest;

    return request;
});

ResponseConverters.Add((req, response) => {
    if (!req.Items.ContainsKey("_encrypt"))
        return null;

    var encResponse = CryptUtils.Encrypt(SecureConfig.PublicKey, response.ToJson());
    return new EncryptedMessageResponse {
        OperationName = response.GetType().Name,
        EncryptedBody = encResponse
    };
});

Request and Response Converts are now available on MyGet.

Thanks for the detailed writeup. Have you also considered using a Hybrid Encryption strategy? as it’s inefficient to use Public/Private asymmetric encryption for encrypting large messages. This answer provides a nice summary of what the process would look like for generating and encrypting messages with a symmetric key.

1 Like
8

Nice, thanks.

I’ll want to use the new RequestConverter and ResponseConverter ultimately. I might need to wait though until this comes with next release.

Hybrid encrypt is good. for now our messages over this channel are tiny, so something to build up to in future, YAGNIing it for now.

Thanks for your extra cycles on this.

9

@mythz one question about the new RequestConverter

Let’s say that the encrypted message cannot be decrypted to a real instance of the type it should be.
Is the RequestConverter the right place to send back a 400 - BadRequest?

Who should do that? RequestConverter or something else? if so how?

10

@jezzsantos Have a look at the new EncryptedMessagesFeature which uses the Hybrid RSA+AES encryption strategy mentioned above, it supports encrypting and returning error responses in both Request and Response converters.

It also includes a new typed EncryptedServiceClient that supports both JsonServiceClient and new JsonHttpClient as seen in the EncryptedMessagesTests

11

OK, so what I glean from that example (for others coming after) is that I should throw ArgumentException in my RequestConverter if the encrypted message is either malformed or the serialization to the specified type fails in some way.

12

@mythz as I am implementing this now fully, it is becoming clear that the vital ‘authenticity’ part of the interaction with an encrypted service is where the most labour and value exists.

You see, up to this point our RequestConverter and your EncryptedMessagesFeature have only been concerned with ‘confidentiality’ of the interaction, and have ignored ‘authenticity’ of the caller. Not sure if you were going to go there, but without authenticity confidentiality is moot, because any attacker can use the public key to encrypt and send a request. Without authenticity the service has no way to know who sent the message, and no way to prevent replay attacks.

What we are doing in our current implementation is to pass the following data in the encrypted message in a required part of the unencrypted dto: UserId, SessionId, Nonce
The RequestConverter ensures that this authenticity data is present in the unecrypted DTO (by interface), and a DTO validator ensures its existence and correct range. The last piece is to actually verify that the UserId, SessionId and Nonce are all correct values.
UserId identifies the caller
SessionId, (negotiated out of band) ensures the caller is who they say they are
Nonce: ensures this request has not been repeated before.

I am thinking that this verification step could be done within each service method, but better (for maintainability, SOC etc), it could be in the scope of the RequestConverter (delegated to another component). Any failures in this verification stage should result in a 401 (I think). Also needs to be backed by storage for the UserId, SessionId and Nonce.

Are you considering either including any authenticity at some point?, or perhaps posting a warning to implementors that all they get with the EncryptedMessagesFeature is confidentiality, not authenticity, and they should carefully roll their own? Thoughts?

13

I don’t agree, IMO the primary value is the encrypted message that only the client and Server can read.

How can it be moot? It allows sending encrypted messages on an unencrypted channel that can’t be read by an attacker. This is the same value normal https provides where anyone can access a secure bank website. The vital part is absolutely that the authentication and communication traffic cannot be read by an attacker.

You’re right that trying to add authenticity brings in a lot more additional complexity, (which IMO devalues the feature) which is why I’ve avoided it. The current implementation is both trivial and easy to understand and just works whilst essentially providing the same value of using https but over http where Users can send a normal Authentication request to setup the Users Session and return the SessionId that can later be used to access Authenticated Services:

var client = CreateClient();
IEncryptedClient encryptedClient = client.GetEncryptedClient();

var authResponse = encryptedClient.Send(new Authenticate {
    provider = CredentialsAuthProvider.Name,
    UserName = "test@gmail.com",
    Password = "p@55word",
});

var response = encryptedClient.Send(new HelloAuthenticated {
    SessionId = authResponse.SessionId,
});

This is conceptually the same as using a Cookie to send the SessionId but instead the SessionId is explicitly sent in the encrypted message which is why it makes sense for ServiceStack and integrates with the existing security model without adding additional complexity.

I understand it doesn’t cover all use cases, but IMO still adds a lot of value, is simple to use and understand and integrates seamlessly with ServiceStack which is why it makes sense as the built-in implementation. I’ll be explicitly mentioning what it does and doesn’t provide when I document the feature.

The implementation with “authenticity” I was considering earlier went something along the lines of maintaining public keys for each User stored in a RDBMS. The User could then sign the message and send a hash of their public key that was used. The Server then checks the hash matches a public key it has registered for that user, if it can verify the signature, send the response back encrypted with a new AES key that was encrypted with the users public key. But this implementation would’ve given the feature a lot less utility since the both clients and servers now have to concern themselves with key management, it’s also opinionated and still likely not support most use-cases for users who have more specific requirements - So I’ve gone with the easiest solution that just works and can provide instant utility that anyone can use at the expense of not cryptographically verifying the client.