SqsMqServer never fires handlers

la2texas · June 27, 2018, 2:41am

I’m running a BasicAppHost because I don’t need any HTTP (only MQ messaging). This service is a windows service that is deployed to various state agencies and acts as a middle man between my normal SS Web API and other state services.

All works great with RabbitMq in development. I can startup a rabbit mq docker instance, setup the config on my Web and BasicAppHost, and “pass” messages from the Web to the BasicAppHost. The Web only acts as a producer for now.

But as soon as I setup the deploy pipeline to AWS, replacing RabbitMqServer with SqsMqServer - nothing fires. I see the Queues created in my AWS console (so I know the BasicAppHost is registering handlers properly), but can’t get messages to fire in my Beta AWS environment, nor on localhost debug mode.

The AWS SQS queues show no messages have ever been received.

Relevant code

Web API Startup

var sqsAccessKey = AppSettings.GetString("SQS:AccessKey");
var sqsSecretKey = AppSettings.GetString("SQS:SecretKey");
container.Register<IMessageService>(c => new SqsMqServer(sqsAccessKey, sqsSecretKey, RegionEndpoint.USGovCloudWest1));

Web API Producer Usage

var envName = AppSettings.GetNullableString("Environment:Name");

var queueName = string.IsNullOrWhiteSpace(envName) || envName == "Local"
    ? sessionUser.QueueName<StateHostRequest>()
    : $"{envName}.{sessionUser.QueueName<StateHostRequest>()}";

if (envName != "Local")
    queueName = queueName.ToValidQueueName();
// Simplified, this comes out to: Beta-LA-mq-StateHostRequest-inq which matches what is in SQS

using (var mqClient = ResolveService<IMessageService>().CreateMessageQueueClient())
{
    var stateHostRequest = new StateHostRequest()
    {
        //...
    };

    mqClient.Publish(queueName, new Message<StateHostRequest>(stateHostRequest));
}

Windows Service Self Host Setup (State Host)
Each state has it’s own host, so we register it with a prefix of [EnvironmentName].[StateAbbreviation].

var sqsStatePrefix = AppSettings.GetString("SQS:StateQueuePrefix");
var sqsEnvironmentName = AppSettings.GetString("Environment:Name");
var sqsAccessKey = AppSettings.GetString("SQS:AccessKey");
var sqsSecretKey = AppSettings.GetString("SQS:SecretKey");

QueueNames.SetQueuePrefix($"{sqsEnvironmentName}.{sqsStatePrefix}");
container.Register<IMessageService>(c => new SqsMqServer(sqsAccessKey, sqsSecretKey, RegionEndpoint.USGovCloudWest1)
{
    DisableBuffering = true
});
var mqServer = container.Resolve<IMessageService>();
mqServer.RegisterHandler<StateHostRequest>(this.ExecuteMessage);
mqServer.RegisterHandler<StateHostResponse>(this.ExecuteMessage);   
mqServer.Start();

I’ve also tried changing BasicAppHost to AppHostBase, and calling mqServer.Start() in my SS Web API (although I don’t believe I need it to simply produce messages, at least, I never did with RabbitMqServer).

How do messages get from AWS SQS to my mqServer? Is it a poll? Does my service need to be on some public IP address? Does it use HTTP?

mythz · June 27, 2018, 4:11am

AwsApps has an example of using SqsMqServer:

github.com

ServiceStackApps/AwsApps/blob/2fef4967f24005f927548dc08d2f1978f08765ef/src/AwsApps/AppHost.cs#L89-L99


private void ConfigureSqsMqServer(Container container)
{
    container.Register<IMessageService>(c => new SqsMqServer(
        AwsConfig.AwsAccessKey, AwsConfig.AwsSecretKey, RegionEndpoint.USEast1) {
        DisableBuffering = true,
    });


    var mqServer = container.Resolve<IMessageService>();
    mqServer.RegisterHandler<EmailContacts.EmailContact>(ExecuteMessage);
    mqServer.Start();
}

Here’s the implementation of SqsMqServer, it effectively uses AWSSDK AmazonSQSClient to endlessly poll the queue waiting for new messages. Here’s effectively where it’s calling AWS SQS to receive messages:

github.com

ServiceStack/ServiceStack.Aws/blob/daea0db6897541d4f52a9db6fb88eef1807d92a3/src/ServiceStack.Aws/Sqs/SqsMqBuffer.cs#L290


public Message Receive(ReceiveMessageRequest request)
{
    if (request == null)
        return null;


    if (receiveBuffer.Count > 0 && receiveBuffer.TryDequeue(out var toReturn))
        return toReturn;


    request.MaxNumberOfMessages = Math.Min(SqsQueueDefinition.MaxBatchReceiveItems, Math.Max(request.MaxNumberOfMessages, 1));


    var response = SqsClient.ReceiveMessage(request);


    return BufferResponse(response);
}


private void NakBufferedReceived()
{
    Message toNak = null;


    var stopAt = DateTime.UtcNow.AddSeconds(30);

la2texas · June 27, 2018, 4:22am

If my setup is correct, any idea why no messages are ever published to my Queue? I’ve checked and rechecked my configuration and usage but I’m baffled. No errors are thrown, just nothing.

mythz · June 27, 2018, 4:30am

I don’t, you can try using AWS SDK’s AmazonSQSClient directly to see if that’s returning anything.

la2texas · June 27, 2018, 4:02pm

Ugh, this is what happens when I stare at a problem too long. It turns out I DIDN’T disable buffering in my main Web API. I believe this was causing a queue to get filled, but never reached a threshold for sending.

One thing I’ve noticed though, is that AWS SQS does not work as well as Rabbit MQ. 80% of my messages go straight to the DLQ (as soon as I call .Publish(queueName, message), it’s in the DLQ). Is there some setting I need to set to change the wait timeout for a message pickup?

la2texas · June 27, 2018, 6:41pm

This is resolved now, thank you for your help.