We are running .Net Framework processing poco messages off of RabbitMQ in a windows service loading the AppSelfHostBase. We are updating to RabbitMQ 5.2 and ServicStack 6.1, just want to make sure this is a good version combination to run together or are there any known issues? When we start we are seeing several worker exceptions raised with the “Received exception in Worker: mq:xxxxxxxx”. Before digging in wanted to make sure there weren’t any known compatibility issues with versions I am using?
No aware of any specific issues with that specific version combination. I assume you mean RabbitMQ client version 5.2 of the NuGet dependency, do you know what version of RabbitMQ you are running? Any details of the exception in the worker you can share?
Here’s an example combination of exceptions getting raised for one service. The handler is registered successfully, but once things are started it throws the following combination for the services when waiting for messages.
2022/11/14 17:52:14.323: Received exception in Worker: mq:UnsubscribeEmail.inq
2022/11/14 17:52:14.819: Interrupting previous Background Worker: RabbitMqWorker: mq:UnsubscribeEmail.inq
2022/11/14 17:52:17.841: RabbitMqWorker: mq:UnsubscribeEmail.inq just wont die, so we’re now aborting it…
2022/11/14 17:52:17.841: Error Disposing MessageHandlerWorker for: mq:UnsubscribeEmail.inq
These look like custom logging messages, do you have any more info or stacktrace from the original exception?
They are exceptions raised from the ServiceStack RabbitMQ source and written out through the logger. They can be located in the source. I can try to get the source and step through things, but wanted to verify there isn’t any known issues with the RabbitMQ and ServiceStack versions.
This is what happens after an Exception is thrown in the RabbitMqWorker or when Disposing (e.g. App is shutting down) where it waits for 500ms for the Thread to complete on its own, if it doesn’t will abort it which is what the error messages are reporting.
So there’ll either be an original source Exception causing the Exception in the worker or if it happens when the App is shutting down.
Spent some time looking into this further. This is happening during the Host Start which is establishing several connections to RabbitMQ. If we have roughly 25 or fewer we don’t get any errors, but if there are more connections than that it starts raising those errors. It seems to eventually load and process messages after a few minutes. We are upgrading to ServiceStack 6.1 and RabbitMQ.Client 5.2. What we found was that we can replace the RabbitMQ library with 5.0.1 our previous version and things load up quickly with no errors. Talking 100+ connections in 1 or 2 seconds vs. 5-10 min. Wondering if I’m needing to set a property with the new libraries, why is this RabbitMQ.Client version different? I tried several of the other client libraries with same results, just seems to be 5.0.1 that’s working well.
Are all other dependencies the same? It is possible the issue with slowness/root cause is else where cause by the upgrade which is causing the timeouts/ThreadInterruptedException.
Here is an example of similar errors/issues but was something further down. Are you using this for just one service/queue or do you have many queues? Are all slow when underload?
My mistake, this is error from the RabbitMqWorker as @mythz stated. Is it possible some other resources are not being disposed of causing this issue? If you can isolate the issue to a reproducible project, we can test and help find a solution.
Also, if you are able to share any code related to what is being processed, post it on a public GitHub gist/project, and we can review to point out possible root causes.
I downloaded the service stack source and setup break points to see where the exception is getting raised. Note that this happens during the startup and isn’t noticeable with a few services (workers). It loops through a number of times with the error but eventually it connects and process messages off the queue. It happens consistently with our services. I haven’t tried adding the source to an example project to see if it happens there. I was hoping someone may have some ideas. Have to turn on logging to see the errors otherwise it would just seem to take a while to startup since the loop eventually seems to get the workers connected to the queues. We have no problem in RabbitMQ.Client 5.0.1, but appears to be a problem after that.
Here’s what I see when ConnectSubscription is called. The following exception is raised to the RabbitMQWorker > Run method’s catch statement.
{“Attempt by method ‘ServiceStack.RabbitMq.RabbitMqBasicConsumer…ctor(RabbitMQ.Client.IModel)’ to access method ‘RabbitMQ.Util.SharedQueue`1<System.__Canon>…ctor()’ failed.”}
Here’s the stack trace.
at ServiceStack.RabbitMq.RabbitMqBasicConsumer…ctor(IModel model) in C:\Dev_SourceRepos\RnD\ServiceStack\ServiceStack\src\ServiceStack.RabbitMq\RabbitMqBasicConsumer.cs:line 11
at ServiceStack.RabbitMq.RabbitMqWorker.ConnectSubscription() in C:\Dev_SourceRepos\RnD\ServiceStack\ServiceStack\src\ServiceStack.RabbitMq\RabbitMqWorker.cs:line 262
at ServiceStack.RabbitMq.RabbitMqWorker.StartSubscription() in C:\Dev_SourceRepos\RnD\ServiceStack\ServiceStack\src\ServiceStack.RabbitMq\RabbitMqWorker.cs:line 209
at ServiceStack.RabbitMq.RabbitMqWorker.Run() in C:\Dev_SourceRepos\RnD\ServiceStack\ServiceStack\src\ServiceStack.RabbitMq\RabbitMqWorker.cs:line 113
Not sure where to go from here, it appears like it could have been an interface change with the RabbitMQ.Client?
Are you running this in an environment with rules around trusted assemblies/services/files? Eg, was the previous version of the RabbitMQ dll trusted/allow listed somehow vs the updated version isn’t?
Have a read of this SO answer and see if you can validate CAS isn’t part of the problem since you are running as a Windows service.