Routing via user id

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Routing via user id

Sean Joyce
I have a scenario where one producer publishes a stream of messages to an exchange. The exchange has 10 queues (there is an upper limit to the amount of parallel work that can be done - e.g. Database bottleneck) that it routes messages to. The messages can really go to any queue with one exception: only one message with a given user id can be processed at a time. 

All messages with that user id should be routed to the same queue. If there was no database bottleneck I would create a new queue for each user id but I can really only process ten messages at a time. 

I had thought of partitioning the user ids across ten queues but there is no guarantee the incoming messages would be evenly distributed even with different user ids. Thus I could have queues 1,4 and 8 with a backlog while the rest are idle resources. 

Any tips on a defined pattern that solves this type of problem so that any message for a given user id can go to any queue but messages with the same user id always go to the same queue?

-Sean

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Routing via user id

Laing, Michael P.
You should look at the 'consistent-hash-exchange': http://www.rabbitmq.com/plugins.html

By either using the user id as the routing key or by setting a header to the user id, messages w a particular user id will always get routed to the same queue.

Hopefully your input is nicely distributed across the user id space. If not, there is more work required.

ml


On Sun, Jul 13, 2014 at 11:41 AM, Sean Joyce <[hidden email]> wrote:
I have a scenario where one producer publishes a stream of messages to an exchange. The exchange has 10 queues (there is an upper limit to the amount of parallel work that can be done - e.g. Database bottleneck) that it routes messages to. The messages can really go to any queue with one exception: only one message with a given user id can be processed at a time. 

All messages with that user id should be routed to the same queue. If there was no database bottleneck I would create a new queue for each user id but I can really only process ten messages at a time. 

I had thought of partitioning the user ids across ten queues but there is no guarantee the incoming messages would be evenly distributed even with different user ids. Thus I could have queues 1,4 and 8 with a backlog while the rest are idle resources. 

Any tips on a defined pattern that solves this type of problem so that any message for a given user id can go to any queue but messages with the same user id always go to the same queue?

-Sean

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Routing via user id

Michael Klishin-2
On 14 July 2014 at 02:48:37, Laing, Michael ([hidden email]) wrote:
> > You should look at the 'consistent-hash-exchange': http://www.rabbitmq.com/plugins.html 
>  
> By either using the user id as the routing key or by setting a header  
> to the user id, messages w a particular user id will always get  
> routed to the same queue.

as long as the list of bindings does not change. The  point of using consistent hashing
is to minimise the number of key/queue "re-mappings" when queues are bound
or unbound, though.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss