ServiceStack on Kestrel holds open files?

We’re hosting a ServiceStack AppHostBase in an instance of Kestrel on an AWS machine instance, and after polling the service every second for an hour or so the Linux machine fails reporting ‘Too many open files’.
So I’ve taken a look at the reported open files for the process, and this is increasing by one for every request processed.
For example:

COMMAND  PID USER   FD      TYPE  DEVICE SIZE/OFF     NODE NAME
:
dotnet  7586 root 1771u     IPv4 2496218      0t0      TCP localhost:40786->localhost:http (ESTABLISHED)
dotnet  7586 root 1772r     FIFO    0,11      0t0  2496221 pipe
dotnet  7586 root 1773w     FIFO    0,11      0t0  2496221 pipe
dotnet  7586 root 1774u     IPv4 2496223      0t0      TCP localhost:40788->localhost:http (ESTABLISHED)
dotnet  7586 root 1775r     FIFO    0,11      0t0  2496226 pipe
dotnet  7586 root 1776w     FIFO    0,11      0t0  2496226 pipe
dotnet  7586 root 1777u     IPv4 2496228      0t0      TCP localhost:40790->localhost:http (ESTABLISHED)
dotnet  7586 root 1778r     FIFO    0,11      0t0  2496231 pipe
dotnet  7586 root 1779w     FIFO    0,11      0t0  2496231 pipe
dotnet  7586 root 1780u     IPv4 2496233      0t0      TCP localhost:40792->localhost:http (ESTABLISHED)
:

These files just keep getting added until the server fails. I don’t know if this is a Kestrel issue or related to the ServiceStack endpoint not closing the output stream in some way.
Has anyone experienced this and found a solution? It’s a bit of a show-stopper for us at the moment.

ServiceStack is just middleware when it’s running in a .NET Core App. Have a look at the solutions for solving the too many files issue in Kestrel in this post:

Yes, I did that Google search, and his solution was to increase the number of files (so you get the error every year instead of week :slight_smile: ).
It turns out the problem was at the calling end, using the JsonServiceClient.
I changed

_client.Get($"internal/events/process").ThenDo(x => _timer.Start());

in a function to

using (_client.Get($"internal/events/process").ThenDo(x => _timer.Start())) { }

It turns out the response wasn’t being disposed, so was held open. I’m positive I hit this issue in the past on a previous project but forgot all about it.

1 Like

Hmm it’s one of the few places we have comments on :). I also removed a number of deprecated APIs which return HttpWebResponse in the v5 breaking window. I guess we missed those ones. Do you think I should deprecate those APIs as well?

The preferred API would be to explicitly specify the HttpWebResponse so it’s visible on the call-site, e.g:

using (_client.Get<HttpWebResponse>("internal/events/process")) {}

Decided on deprecating and adding more comments.

Sorry, I missed this.
Yes, I agree. It’s just too dangerous to leave floating around in web situations, and the number of ‘fixes’ out there involving increasing the number of files is just scary :smile: