Caching in-process service requests

The ability to cache in-process requests would be very useful to our upcoming project. We plan on using a “layered” approach to building our API (similar to MuleSoft), where low-level services will return raw data from the data store and higher level services will use the base data for aggregations and modeling the output into the final form that will be served to the UI.

The first version of our site does all the aggregation/data shaping work in stored procedures, which makes for a great deal of duplicate logic and cases where the data in two services should match but doesn’t because the logic in 2 stored procs is slightly different.

In v2 of our API, multiple services will ensure using the same base data via In-process, cached, lower-level services.

I plan on implementing a new attribute for cacheable services, the one difficulty I see is determining the cache key for the requests. An idea I had for this is an ICacheKey interface on the DTO.

I was interested in feedback on my idea, and any other problems I might run into from those who know the SS code base better than I do.

If it’s per Request DTO, you may want to consider just using a ConcurrentDictionary.

To calculate the cache key for the Request DTO you basically need to capture all the information that makes the request unique. If that’s all the properties, then you could serialize it to JSV or JSON and use the Request DTO itself for the cache key.

Alternatively you could generate Equals/GetHashCode for your DTOs which R#/JetBrains Rider can easily generate this with its Generate > Equality Members feature.

I don’t see this as a ServiceStack issue, sounds like you’re just caching the response against the Request DTO, the challenge would be to implement it properly, including cache expiration and invalidation.

The ConcurrentDictionary looks like it would work, what I’m having trouble with is the implementation. I found the GatewayRequestFilter and GatewayResponseFilter, but I’m not sure the best way to short-circuit the requestfilter if the item is found in the cache. I tried to write the object to the response stream but a null is returned from Gateway.Send. Here is pseudocode for the 2 filters.

this.GatewayRequestFiltersAsync.Add(async (req, o) =>
{

    theConcurrentDictionary.TryGetValue(req.CacheKey, out var item);
    if (item != null)
    {
        var res = req.Response;
        res.StatusCode = (int)HttpStatusCode.OK;
        res.ContentType = MimeTypes.Json;

        await res.WriteToResponse(req, item);
        await res.EndRequestAsync(skipHeaders: true);
    }            
});

this.GatewayResponseFiltersAsync.Add(async (req, o) =>
{
    theConcurrentDictionay.Tryadd(req.CacheKey, o)
});

I believe that I have uncovered a bug when attempting to use the Gateway along with a GatewayRequestFilter to change the response. I have a reproducible example at the link below. The README.md contains instructions for recreating the error.

In short, when you try and change the response in the GatewayRequestFilter, the Gateway always returns null. The code in the filter is similar in nature to the code I posted in the last message of this thread.

The example code can be found here.

thank you

Thanks for the reproduction, makes it super clear what the issue you are seeing is, so much appreciated!

Currently the default behavior won’t return what is populated in the response, so that is something we can look at supporting.

In the meantime, you can use your own IServiceGateway to handle this kind of behavior. To do so, you will need to implement a few interfaces and add some dependencies to your IoC.

Custom InProcessGateway

public class MyInProcessGateway : InProcessServiceGateway
{
    public MyInProcessGateway(IRequest req) : base(req) { }

    protected override async Task<TResponse> ExecAsync<TResponse>(object request)
    {
        if (Request is IConvertRequest convertRequest)
            request = convertRequest.Convert(request);

        var appHost = HostContext.AppHost;
        var filterResult = await appHost.ApplyGatewayRequestFiltersAsync(Request, request);
        if (!filterResult && !Request.Response.IsClosed) 
            return default;
        
        // If req.Response is closed, assume populating the response is handled by the filter
        if (Request.Response.IsClosed)
            return await ConvertToResponseAsync<TResponse>(Request.Response).ConfigAwait();

        if (request is object[] requestDtos)
        {
            foreach (var requestDto in requestDtos)
            {
                await ExecValidatorsAsync(requestDto);
            }
        }
        else
        {
            await ExecValidatorsAsync(request);
        }

        var response = await HostContext.ServiceController.GatewayExecuteAsync(request, Request, applyFilters: false);

        var responseDto = await ConvertToResponseAsync<TResponse>(response).ConfigAwait();

        if (!await appHost.ApplyGatewayResponseFiltersAsync(Request, responseDto))
            return default;

        return responseDto;
    }
}

Here, the main change is the population of the result if the req.Response is closed.

var filterResult = await appHost.ApplyGatewayRequestFiltersAsync(Request, request);
if (!filterResult && !Request.Response.IsClosed) 
    return default;

// If req.Response is closed, assume populating the response is handled by the filter
if (Request.Response.IsClosed)
    return await ConvertToResponseAsync<TResponse>(Request.Response).ConfigAwait();

Custom IServiceGatewayFactory

public class MyInProcessGatewayFactory : IServiceGatewayFactory
{
    public IServiceGateway GetServiceGateway(IRequest request)
    {
        return new MyInProcessGateway(request);
    }
}

This enables the ability to register and override the default InProcessGateway.

Register dependencies within IoC

Depending on which IoC you are using, the syntax might be a bit different, but here is the code for registering against the built in .NET 8 IoC.

public class AppHost() : AppHostBase("GatewayTest"), IHostingStartup
{
    public void Configure(IWebHostBuilder builder) => builder
        .ConfigureServices(services => {
            // Configure ASP.NET Core IOC Dependencies
            services.AddTransient<IServiceGatewayFactory>(provider => new MyInProcessGatewayFactory());
        });

The code in your example for your GatewayRequestFiltersAsync stays the same.

        this.GatewayRequestFiltersAsync.Add(async (req, dto) =>
{
    if (dto is HelloGateway hello && hello.Mode == 2)
    {
        var res = req.Response;
        res.StatusCode = (int)HttpStatusCode.OK;
        res.ContentType = MimeTypes.Json;

        var responseDto = new HelloGatewayResponse 
        { 
            Result = $"Hello, {hello.Name}, short-circuited in the Request Filter" 
        };

        await res.WriteToResponse(req, responseDto).ConfigAwait();
        await res.EndRequestAsync();
    }
});  

And the request is short-circuited.

image

Hope that helps.

Is there any way you could send me that working example? That would be awesome

Appreciate the support - I’ve been a happy customer for over 10 years and the product just gets better and better

I’ve shared it as a GitHub repository, so you should be able to clone and run it.

thank you, it’s working great.

@taglius This functionality is now included by default in this commit, and is now available via our pre-release.

1 Like

thanks must for your help so far. I would like to report one strange thing with the solution, perhaps it is intentional or can’t be avoided. When the GatewayRequestFilter successfully short circuits and writes the response bytes, the return value from Gateway.Send is always null. So in the code below (from my sample program), the short-circuited object is returned onto the web page from the response stream, but grsp is null

var grsp = base.Gateway.Send<HelloGatewayResponse>(
    new HelloGateway
    {
        Name = request.Name,
        Mode = request.Mode
    });

var rsp = new HelloResponse { Result = grsp?.Result ?? "error occurred in GatewayRequestFilter"};
return rsp;

If I set a breakpoint on the var rsp =... line, I can see the correct short-circuited returned value in base.Gateway.Request.Response.Dto, but that object is not returned to variable grsp

Mistake on my part, since by default ASP.NET doesn’t enable the ability to read from stream, you won’t be able to get the response back using the code above, so what it is returning is actually the NetCoreResponse rather than the response DTO, which is why the grsp.Result is null. The conversion returned a HelloGatewayResponse but couldn’t map any results. So this approach is likely not viable since enabling request rewind (to support reading the response stream) isn’t very common.

In your situation where you want to cache results, instead of using a Gateway filter, handling this within your own custom IServiceGateway might be a better approach, rather than the gateway directly modifying the response stream. Eg, the gateway is where you would manage the logic for how you want to cache/fetch the ‘raw data’ and hand it back to your common more high level services. This will be more flexible to handle future changes as well where if how your data is stored changes, only your gateway Exec logic will need to manage the changes.

of course! I’ve been looking for a point to run cache.GetOrCreate(), I can do it right in the Gateway!