LastModified Headers Missing after moving to dotcore on linux

.NET Core
1

So we just recently moved our services from an IIS hosted .net framework 4.7 to linux hosted dotcore. There really wasn’t much change in doing so. However, we have since noticed that our LastModified headers sometimes work and sometimes don’t. We’ve got nothing more to go on and looking for maybe some ideas or direction on where to look.

Is there default configurations that are different between the hosting setup that would affect it? Or maybe because we have ngix in front of it? any thoughts?

Update - one of the developers things something in the cachcontrol options? They are not returning the same values between IIS version and dotcore version. We are getting private vs no-cache being returned.

2

It’s using the same Kestrel HTTP Server everywhere so there shouldn’t be any difference, so I’m assuming it’s related to the reverse proxy. Do you have a side-by-side of the HTTP Headers of the same request in Windows/Linux?

3

So it’s not the same kestrel server unless that’s something under the hood. We use PostApplicationStartMethod to start the IIS app host in the .framework version. It was rather old and it’s own project.

The AppHost is mostly the same with a few small changes to how we loaded some parameters into the HostConfig. The old used web.configs and the new is just hard coded right now.

I’ll work with my dev to get the side by side headers shortly.

4

.NET Core uses the same cross-platform Kestrel HTTP Server from:

Whether it’s running behind an IIS or nginx reverse proxy it still uses the same implementation.

5

image (4).png1920×629 98 KB

6

What are the headers if you request the .NET Core HTTP Server directly, i.e. without going through nginx?

7

Sorry, don’t have an image right now, but they are the same. Further investigation shows the problem also shows up in IIS and started for us last week. I will take some time tomorrow to start going back through commits. The one major change was that we took our API project which was only pure framework, broke out the services into their own assemblies (converted to core) and then added a new API project for core with same apphost (few changes).

It appears to be somewhere at that time when we release to dev (still as IIS site framework version) that we started noticing it.

8

So I can only come to the conclusion right now that something in kestrel/core apphost is different from the IIS/non-kestral .
My IIS app version build on framework 4.7 with the same SS version will return my headers for Last-Modified as well as 304 status on successive calls. The core version omits the Last-Modifed header and returns 200, not 304 as expected.

9

The Kestrel used is the same, so it’s likely due to IIS behavior, a workaround posted:

I’ve found a workaround, enable caching and then disable it on an extension level did the trick for me.

<configuration>
  <system.webServer>
    <caching enabled="true" enableKernelCache="true">
      <profiles>
        <add extension=".png" policy="DisableCache" kernelCachePolicy="DisableCache" />
        <add extension=".gif" policy="DisableCache" kernelCachePolicy="DisableCache" />
        <add extension=".js" policy="DisableCache" kernelCachePolicy="DisableCache" />
        <add extension=".css" policy="DisableCache" kernelCachePolicy="DisableCache" />
      </profiles>
    </caching>
    <staticContent>
      <clientCache cacheControlMode="NoControl" />
    </staticContent>
  </system.webServer>
</configuration>

Another reported issue from iis forums suggests it’s a bug in handling E-tags:

It has been noticed that if the ETAG (If-None-Match) is not present in the header, then the ARR server returns Http304.

 trying to figure out how/why ETAG causes this issue in ARR..

 

Looks like its better to disable ETAG from IIS completely on Web Farms (including Reverse Proxies with ARR):
10

Sorry, maybe I’m not being clear. I do expect the 304 response on multiple calls to the endpoint, but I’m not getting that. Additionally, the version that works is running on IIS, not Kestrel.

I’m going to try and get a small solution together which can hopefully replicate the problem to try and narrow it down.

11

Ok but is the issue with Kestrel or nginx? i.e. are the right headers emitted when calling .NET Core Kestrel server directly (instead of going through nginx)?

12

nope. It’s not returning 304 when expected. It’s returning 200, with no Last-Modifed header. (the first call will return 200 with last-modified set correct).

But, I suspect this is because the Cache-Control is being changed and successive calls are not hitting the api and thus no last-modified.

I’m just working through some testing against our last production released code (IIS version) that is working. It’s going to take a bit as there are some setup things that need to be modified to test locally.

13

Now I’m even more confused and frustrated than ever. I’ve pulled our release branch that’s currently running fine in production and have it running locally. It’s experiencing the same issues as our development branch. It’s running on IIS.

In my AppHost we have the following setup:

SetConfig(new HostConfig
        {
            EnableFeatures = Feature.All.Remove(GetDisabledFeatures(container.Resolve<Settings>())),
            ApiVersion = "2.3",
            GlobalResponseHeaders = new Dictionary<string, string> {
                { HttpHeaders.Vary, $"accept, origin, authorization, {ApiCustomHttpHeaders.UserApiKey}, {EditLimitHeader}" },
                { HttpHeaders.CacheControl, "no-cache" },
            },
            DebugMode = true
        });

and

  Plugins.Add(new CorsFeature(
            allowedHeaders: "Content-Type, Authorization, If-Unmodified-Since, If-Modified-Since, X-ss-opt, X-ss-pid, X-ApiKey",
            allowCredentials: true,
            exposeHeaders: $"{EditLimitHeader}, Content-Disposition",
            allowOriginWhitelist: originList.AllowValues

        ));

You can see we explicitly set no-cache. However, on the first call to the endpoint the Cache-Control is coming back private.

image (6).png873×624 52.9 KB

I make another call to a POST endpoint that modifies the record where I should then be able to call the original GET again to get an updated record with new Last-Modified value. However, I get a load from cache due to the Cache-Control being set private instead of no-cahce.

image (7).png882×561 47 KB

14

If you’re just using GlobalResponseHeaders to set cache headers (i.e. aren’t using them in addition to cache features) then I’m assuming IIS is modifying the value. What are the headers from the .NET Core host directly?

15

Well after many many many hours it seems we’ve figured something it out. It appears to be due to SynchronizationContext. See here.

Why it’s also the case in my release branch that is working in production is a mystery still.

Many of our endpoints have been Tasks for a very long time. But, in the last 2-3 months we have had many strange issues start cropping up with Task endpoints that had not been touched in months. Some would totally block, others would return random exceptions and some would return different results than they should. There was no reason it seemed other than they were Tasks. Yet, many Task endpoints continues to work fine. In fact, we still have many that are running fine, yet it’s a particular few that have been causing the cache/last-modified issues above this time.

We changed the endpoints to use Task.Wait and not be defined as async endpoints and everything started working as expected. I’m still not fully understanding why and will have to do some additional research. This was found and fixed by one of the others on my team.

1 Like
16

So to further the discussion here there are some additional things to note.

First, if you have an endpoint that calls another existing endpoint as follows:

  public IHttpResult NewGet(DetailRequest request)
    {

            var pgs = ResolveService<ProposalService>();
            return pgs.Get(request);
        }

The last-modified header set in the prs.Get function doesn’t get passed through. It’s stripped off for some reason. That function is simply

 return new HttpResult(response, HttpStatusCode.OK)
        {
            LastModified = response.ModifiedDate
        };

But, if you modify to call that function directly (it’s the one resolved by servicestack) the header is correct. So this confuses me.

Additionally, what we are now seeing is the response.ModifiedDate isn’t being treated as UTC (it was previously) and the value being 2018-09-26 1:13:45 AM is being set in the header as Wed, 26 Sep 2018 05:13:45 GMT.

17

Cache Headers are ignored for InProcess Requests. If you resolve a Service you must dispose it to release the In Process state, i.e:

using (var pgs = ResolveService<ProposalService>())
{
    return pgs.Get(request);
}

I don’t know what this means but LastModified is written out as UTC:

18

Thanks, I had no clue about that in process caveat.

So for LM, our date was Unspecified kind. Previously it worked, but now I have to make sure i specify the kind as UTC before it gets set to LM. Not sure what changed.

19

Conclusion and final thoughts…

The root of the problem stemmed from how we were calling the InProcess request. It’s a bit frustrating as how is one suppose to know. At the same time I understand why that would be the case and that we really should have been disposing of it in the first place. It just made this so hard to track down.

It was complicated more by the DateTime issue, which it turns out we’ve only been getting away with by pure luck. Our deployment servers were all set to UTC timezone, and thus things worked previously. Now, on linux, the server had a timezone set, complicating the issue. Additionally, unit tests didn’t catch this and because of how we develop, it’s not something we noticed locally either - until we did and just assumed it was never an issue before.

20

Everytime you resolve any dependency from the IOC yourself you need to dispose it if it’s disposable, which all Services are. Pretty sure everywhere that shows ResolveService used to call Services it’s in a using statement in Service Gateway and in Authentication. Where did you find out about ResolveService? If it’s in docs or one of my answers/comments I’ll update it with the correct usage.