Can we run Memcached on dotnet core? Having Redis issues

ServiceStack.Redis
21

Ok. Some more updates. I changed to the RedisPoolManager again now that our connections exceeded issue was resolved by changing within redis. We stopped seeing redis connection pool issues after an API restart (which still occurred).

Additionally we added settings to reduce the amount of parallel processing which we suspected to be the CPU spike. CPU peeked at less than 70% after those two changes.

However, we continue to see the API locking up and restarting. So kind of back to square one there.

Additionally, what we did see happening much more often was the zero length response errors from redis, which all occur while trying to fetch custom user session with SessionAs<> in the base Service class.

After repeated attempts to repeat the error locally, I managed to do so, only once. What I noticed is that the exception came from another thread. So as you alluded to. However, here’s where I get a bit confused.

It was coming from the authentication attribute. The class has a custom auth attribute, which is as follows:

  public sealed class HmacOrCredentialsAuthRequiredAttribute : RequestFilterAttribute
{
    private static readonly ILog Logger = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

    public HmacOrCredentialsAuthRequiredAttribute()
    {
        Priority = -901;
    }
    public override async void Execute(IRequest req, IResponse res, object requestDto)
    {
        if (HostContext.AppHost.HasValidAuthSecret(req))
        {
            req.Items.Add(AuthSignatureRequiredAttribute.ApiUserKey, new ApiUserKeys() { UserId = req.Headers[ApiCustomHttpHeaders.UserApiKey].ToInt() });
            return;
        }

        if (!res.IsClosed)
        {
            if (ApiSignature.HasHmacHeaders(req))
            {
                var auth = new AuthSignatureRequiredAttribute();
                auth.Execute(req, res, requestDto);

            }
            else
            {
                // we want to check authenticate (login/session)
                var auth = new AuthenticateAttribute(base.ApplyTo);
                await auth.ExecuteAsync(req, res, requestDto);
            }
        }
    }
}

Now, I was already inside of the endpoint which required the auth check and no other endpoints were being called.

The exception came from the auth.ExecuteAsync call.

 ServiceStack.Redis.RedisResponseException: Zero length response
  at ServiceStack.Redis.RedisNativeClient.ParseSingleLine(String r)
  at ServiceStack.Redis.RedisNativeClient.ReadData()
  at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
  at ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs)
  at ServiceStack.Redis.RedisNativeClient.GetBytes(String key)
  at ServiceStack.Redis.RedisNativeClient.Get(String key)
  at ServiceStack.Redis.RedisClient.GetValue(String key)
  at ServiceStack.Redis.RedisClient.<>c__DisplayClass131_0`1.<Get>b__0(RedisClient r)
  at ServiceStack.Redis.RedisClient.Exec[T](Func`2 action)
  at ServiceStack.Redis.RedisClient.Get[T](String key)
  at ServiceStack.Redis.RedisClientManagerCacheClient.Get[T](String key)
  at ServiceStack.ServiceExtensions.GetSession(IRequest httpReq, Boolean reload)
  at ServiceStack.AuthenticateAttribute.ExecuteAsync(IRequest req, IResponse res, Object requestDto)
  at HmacOrCredentialsAuthRequiredAttribute.Execute(IRequest req, IResponse res, Object requestDto) in HmacOrCredentialsAuthRequiredAttribute.cs:line 36
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c.<ThrowAsync>b__7_1(Object state)
  at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(Object state)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Threading.QueueUserWorkItemCallback.ExecuteWorkItem()
  at System.Threading.ThreadPoolWorkQueue.Dispatch()
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

So, my confusion is how are we in the calling endpoint yet still not finished with authentication call? What am I missing?

22

I don’t understand the confusion, the StackTrace shows the call stack, which is inside the async state machine that calls the AuthenticateAttribute which is called from HmacOrCredentialsAuthRequiredAttribute.

But you’re using the wrong filter and you should never have async void in your code-base, if you need to use async in your filter you need to inherit RequestFilterAsyncAttribute (which is also what AuthenticateAttribute uses) and override ExecuteAsync() method instead, i.e:

public class HmacOrCredentialsAuthRequiredAttribute : RequestFilterAsyncAttribute
{
    public override async Task ExecuteAsync(IRequest req, IResponse res, object dto) 
    {
        //...
    }
}
23

Well, I guess I was confused as to how we could be authenticated and debugging the endpoint but not yet done the authentication call.

However, I assume now it’s because of the async void oversight. I think we originally wrote this before you made the switch to having ExecuteAsync and whomever modified the code didn’t fully understand what was going on.

I’m going to make the change and hopefully things improve. Thanks for the 10th set of eyes and pointing that out.