MQ issues and potential improvements (apologies for relatively long post, do read if you are or plan to use MQ)
We’re looking into integrating message queuing due to its undeniable advantages vs direct communication between services.
We hesitated to implement Redis based solution due to lack of skills of Linux administration but in the end the actual implementation of the MQ Server should not matter much if we are to use built-in support in ServiceStack which nicely hides the actual details behind its own set of interfaces.
Once RabbitMQ support became available, seeing how it’s much easier to install RabbitMQ on Windows, we went that route.
However, there are several things that aren’t so great in the RabbitMQ implementation and some are even show stoppers for us:
1. The threading approach IMHO isn’t great - there’s one thread per queue, and 4 queues per DTO, which means that for moderately sized deployment where services exchange a couple dozen messages (say 20) there’ll be 80 threads(!) created. Now, this isn’t a big issue when self hosting but when in IIS, this just feels wrong. That said, we can live with this for the moment.
2. There are subtle threading issues when sending a message, then synchronously waiting for response - if the service that should handle the sent message isn’t running (rare occurence but possible), the requester will send another message meant for the same non-running service (while the previous request is still being ‘handled’) but then the underlying library RabbitMq.Client will throw “Pipelining of requests forbidden”; based on some vague explanations in the rabbitmq forums, this appears to be an ‘error’ in the code using the library, which in this case would be ServiceStack; I am not entirely sure that their reasoning is correct, but if it is then it’s a bug (should I post this to Issues?)
3. The main issue for us is not a bug but missing functionality which we hope should be very interesting to anyone using MQ: multi-tenancy. What we mean by this is simple: imagine a database server (say, SQL Server). You most likely have one instance running per environement (test, staging, production). On top of that, each server manages several databases (could be for storage of different applications or same schema but one database per your client). We basically need the latter - to ‘group’ message queues so that we can encapsulate them as one would tables in a database (at the moment, if we wanted separate group of queues, we’d need to install another instance of MQ server). I know that this is generally handled via ‘topics’ or other similar concepts in MQ servers, but this would have to be propagated all the way back to ServiceStack’s implementation and be exposed in interfaces. From the service point of view, it can be surfaced even as a simple modification of the ‘connection string’ so that it includes not just the name of the machine (and port) but also a ‘name’ of… not sure how we would call this, but I hope you understand.
Now, all that said, end of last week Microsoft has shown that they aren’t neglecting the Windows port of Redis, so we’ve started reevaluating using that as an alternative. Issues 1 and 2 from the list above shouldn’t apply, but 3 still does. There’s one more thing I’m not sure about with Redis: does it support acknowledging messages?
Finally, questions: would you consider implementing 3 in the near term? Do you need more info on 1 and/or 2?
Thanks again for the wonderful library, it makes our coding a pleasure each and every day.
Drazen Dotlic:
Additionally, I am aware we can implement whatever we want ourselves by using RabbitMq.Client or any Redis C# client out there (ServiceStack’s or not) but it would be shame to reinvent hot water when ServiceStack’s support is already great. I forgot to say but our glue code which enables service-to-service communication through the MQ is - thanks to ServiceStack’s support - ridiculously small, something like 10 lines of code!
Nicklas Laine Overgaard:
+Drazen Dotlic I have taken the liberty of integrating a client’s MQ code with the MS Redis server port on Windows 2008r2 - so far we’ve seen NO problems at all with Redis. It’s not a high-concurrency enviroment, so the Redis server is not handling more than a 30-50 messages per minute (when it’s wild). I’m using these redis binaries:
https://github.com/MSOpenTech/redis/blob/2.6/bin/release/redisbin64.zip
and then this “redis as a service wrapper”:
https://github.com/downloads/kcherenkov/redis-windows-service/RedisService_1.1.exe
edit: and it’s been running for 6 months now by the way, without misbehaving even once.
Drazen Dotlic:
+Nicklas Overgaard Thanks for sharing your experience. For our uses, we also won’t have a lot of messages passing through (though we will have much more than 50 per minute) but we do need it to be reliable. The reason I’m (re)evaluating Redis is because now that Microsoft is offering it as a part of Azure, they are proving they are definitely committed to continuing to maintain the port. Additionally, our message queueing needs are simple so Redis should be good enough (assuming the point 3 of my post is implemented in ServiceStack), IOW we don’t exactly require a full blown MQ server for our needs, plus we could reuse Redis instance as a shared cache, already supported in ServiceStack (of course). It’s a win-win-win I guess 
Nicklas Laine Overgaard:
+Drazen Dotlic You’re welcome! Stable it has been I must say. And thanks for mentioning the fact about Azure and Redis, i hadn’t noticed that - it’s very good news indeed.
Wayne Brantley:
ServiceStack does not support RPC for RabbitMq. Other alternatives such as http://easynetq.com/ do support RPC. I think Demis is considering supporting RPC based on user requests/feedback though.
There are quite a few other drawbacks to RabbitMQ, so we are considering the use of ServiceBus for Windows (onpremise)
Stephen Patten:
+Wayne Brantley Are you sure? There are tests in the ServiceStack which seem to indicated support for RPC?
Wayne Brantley:
According to my email from Demis on 3/13 it does not and he said it will not be prioritized until there is a demand to it. It could have changed and I missed the update?
Here is his full response:
As always, the place to request features is on ServiceStack;s UserVoice site:
http://servicestack.uservoice.com/forums/176786-feature-requests
It wont be something that’s prioritized until there’s demand for it and the benefits/use-cases for doing so are clear.
RPC-style communication is better done with HTTP where the response is returned as part of the same Request / TCP connection.
A JavaScript RPC/MQ method as you describe would not be possible without utilizing some form of bi-directional HTTP (e.g. using WebSockets) which requires some intermediary for processing MQ responses, correlating them with the request and ensure the response is published back (via bi-directional HTTP) to the right consumer that made the request. This adds additional moving parts/machinery, complexity, etc for processing a service request. Which is contrary to our goal is to reduce complexity and eliminate un-necessary moving parts.
Although the MQ servers already support the Request/Reply pattern by specifying a ReplyTo property:
https://github.com/ServiceStack/ServiceStack/blob/master/tests/ServiceStack.Server.Tests/Messaging/MqServerIntroTests.cs#L161-L171
There are a few other concerns you have to consider when implementing the MQ Reply pattern potentially dependent on your use-case, do you need a temporary unique queue name? unique per request or per consumer? should it use a different priority? should you use a centralized queue and a mediator to correlate requests with their responses? In that case maybe using the outq topic would be preferable? Maybe directly sending the response to a HTTP Service would be better? How long should you block for? should you retry when the operation times out?
I don’t believe these opinions should be wrapped in a synchronous 1-method block, it’s more meaningful to keep the parts separate so it’s clear what’s going on, and if any of it fails, it’s clear where. Others can also choose to implement a more efficient pattern for their use-case, e.g. if I wanted to implement bi-directional MQ reply’s over Ajax I would likely go with a central server processing all MQ Ajax Responses that handles all web-socket client connections and utilize ReplyTo HTTP responses directly to this server that includes the client connection id and operation name so the server can just publish the JSON response to the browser’s web socket connection, the JS client would publish this as an event on the page that multiple subscribers can register callbacks to. This is easily a separate OSS project, maybe it would make a great OSS project for someone in the community who wants to add this functionality to their solution. Either way it would be a fair bit of work and R&D and not something I would prioritize above everything else until there’s enough demand for it.
- There is only 1 thread per .inq or .priorityq (.dlq / .outq do not have listeners). If you’re not using priority queues you can disable them with
mqServer.DisablePriorityQueues = true(mentioned in docs https://github.com/ServiceStack/ServiceStack/wiki/Rabbit-MQ#starting-the-rabbit-mq-server) - leaving only 1 thread per .inq.
2) If you can provide a repro on SS/Issues I can investigate.
3) You can add site-specific namespacing with the QueueNames withQueueNames.SetQueuePrefix("site1."), or completely change the QueueNames used withQueueNames.ResolveQueueNameFn, see the MqNameTests for some examples:
https://github.com/ServiceStack/ServiceStack/blob/master/tests/ServiceStack.Server.Tests/Messaging/MqNameTests.cs
Note: as to be expected when using modified naming, both the server and client need to have the same configuration.
RedisMQ doesn’t have ACK/NAKs, we just use a normal redis server-side list to implement the queuing and a redis topic for notifications of new msgs. Use RabbitMQ if you need ACK/NAKs.
Drazen Dotlic:
+Demis Bellot thanks for your response, precise and useful as usual. A couple observations though:
1. We’re actually already disabling priority queues, yet I thought I saw more threads in the debugger window than just one per .inq… Either way, this was less important than other two issues (and if I do see more threads, will send you a screenshot).
2. I knew you’d say that Fair point, I’ll see if I can repro with a tiny test case.
3. Excellent, didn’t know this even though I’ve gone through the docs (clearly not looked good enough). I’m not sure what you mean by both server and client need to have same configuration - do you mean both server and client side of the relation when exchanging a message? When I say server I usually mean an instance of the MQ server itself, that’s why I’m asking.
We actually don’t require ACK/NAKs, though it would be nice to have.
Wayne Brantley:
It also looks like you would like to use the RPC functionality of MQ, which SS does not support. I have asked for it in uservoice, Demis says to vote on it if you want it. http://servicestack.uservoice.com/forums/176786-feature-requests/suggestions/4459053-add-more-mq-options
As in if you’re using custom mq naming, e.g QueueNames.SetQueuePrefix("site1."), then the client needs it so it sends it with that prefix and the server needs the same configuration so it knows to listen to the prefixed queues.
Drazen Dotlic:
+Demis Bellot Yeah, of course, it’s understood (was just the word server that confused me), thanks.
Drazen Dotlic:
+Wayne Brantley Well, as for RPC (though I wouldn’t call it that) it’s already supported. You can post a message to a queue, then synchronously wait for a response and it will work like a charm, nothing extra is needed. Apart from the threading issue I hint at in my point 2, it works great for us. The client doesn not know who or how processes the message (it turns out both client and server are SS services) so we’ve achieved decoupling we wanted.
Though it would be nice to have a better async support - ideally using async/await rather than today’s “polling” async, but this is a ‘nice to have’.
I guess for people with more complex needs supporting a fuller set of MQ features like RabbitMQ does could be benefitial, for us SS messages service abstraction is currently good enough (especially if we go back to Redis).
Stephen Patten:
+Drazen Dotlic I’d like to thank you for for these very informative and thorough Message Queue posts. I had been struggling with the RabbitMQ implementation vs. the RabbitMQ documentation and originally discounted RPC. Well low and behold it turns out that some of our API clients need synchronous calls to be made, so the RPC pattern seems like the right fit. Again, thank you for laying out the foundation for us.
Were there any other road-blocks that hindered your adoption of RabbitMQ and the ServiceStack abstraction over it’s API?
-Stephen
Drazen Dotlic:
+Stephen Patten You’re welcome. It’s the least I can do seeing how a lot of ServiceStack goodness is thanks to the efforts of the community, I can at least give back some (hopefully) useful feedback.
That said, I’m afraid we had to abandon (temporarily) ServiceStack + RabbitMQ due to two reasons:
1. When you host multiple services each in its own Web app in IIS, there’s a deadlock somewhere in the ServiceStack which blocks the synchronous call. This was a long standing problem that I’ve never gotten to properly repeating in an isolated test case - when I do, I’ll post a bug to the ServiceStack issue tracker.
2. ServiceStack (at the time, a dozen or so revisions back) depended on a relatively older version of RabbitMQ so some of the authentication changes in RabbitMQ broke things in a way not possible for us to fix (requires changes to ServiceStack). Again, once I can trivially repeat issue [1] I’ll ask for ServiceStack to update its dependency for this to be resolved.
There’s a silver lining though - thanks to the fact that MQ is hidden behind interfaces (like everything else in ServiceStack), we were able to switch to the in-memory MQ (this required hosting all services in a single Web app in IIS but again, this is trivial with ServiceStack) with a single line of code and have been using this for the last few months, waiting for the proper chance to revisit the issues and move to proper MQ.
The Request/Reply MQ pattern is both supported and documented: https://github.com/ServiceStack/ServiceStack/wiki/Rabbit-MQ#responses-from-messages-with-replyto-are-published-to-that-address
My reply was in response to your proposal for building in opinionated Ajax RPC/MQ-backed solution. As mentioned above HTTP is superior, simpler and faster for consuming Request/Reply services via Ajax.
Wayne Brantley:
I am sorry if I am confused here. I was speaking of RPC scenario. Not really 'request/reply;, but RPC over RabbitMQ.
Not sure what “RPC over RabbitMQ” means. What’s missing in Request/Reply that’s possible in RPC?
Wayne Brantley:
Ultimately we should be able to make a call something like this: someService.SomeFunction(12) or
var request = new TestRequestMessage {Text = "Hello from the client! "};
bus.Request<TestRequestMessage, TestResponseMessage>(request, response =>
Console.WriteLine(“Got response: ‘{0}’”, response.Text));
The client code would not proceed until that function returns (like normal). That call goes out over RabbitMQ, or whatever. Ultimately I was looking for something that was that simple. Just like what is show at the first example on this page: http://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html
I think it should be that easy and that buttoned up…of course if I want to do it all manually - I could write all that code.
Sorry if I was unclear, have missed something or misrepresented your very nice product - just trying to be helpful. :-)