Introducing the RabbitMQ Sharding Plugin

[Alvaro Videla-2]

Hi,

We have just released the RabbitMQ Sharding Plugin:

The plugin allows you to partition messages across sharded queues.
Adding new nodes to the cluster, automatically scales the shard size.

Learn more here: https://github.com/rabbitmq/rabbitmq-sharding

Any kind of feedback or questions about the plugin is welcomed.

Regards,

Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Gavin M. Roy-2]

Looks great, looks like I have to update another chapter ;-)

On April 10, 2014 at 10:49:51 AM, Alvaro Videla ([hidden email]) wrote:

Hi,

We have just released the RabbitMQ Sharding Plugin:

The plugin allows you to partition messages across sharded queues.
Adding new nodes to the cluster, automatically scales the shard size.

Learn more here: https://github.com/rabbitmq/rabbitmq-sharding

Any kind of feedback or questions about the plugin is welcomed.

Regards,

Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[mc717990]

Off hand, this new exchange looks pretty interesting.  We sorta do sharding via x-consistent-hashses right now.  It does require more manual work/setup to do it, but it works quite well for our setup.  This though looks like it'd make things much simpler - particularly on the consumer side!  A few things I wanted to check on though:
  Shoveling config - we use shovel to publish from a bunch of "cloud" rabbit's to a central server.  We "shard" our queues to get around WAN latency.  SO I'd be curious how this will operate with shovels.  I'm assuming just like any other consumer - e.g. "amqp://@localhost", consume from queue "images"? 
  The one thing I didn't see in the post was how many consumers you'd need.  It sounds like you'd need at least N consumers where N is the number of nodes in the shard?  With the shoveling config, we'd need to dynamically grow consumers to adapt to the cluster size it sounds like.

Just a few thoughts right now - the consumer side is where I've got some concerns on how this would work,

Jason

On Thu, Apr 10, 2014 at 10:17 AM, Gavin M. Roy <[hidden email]> wrote:

Looks great, looks like I have to update another chapter ;-)

On April 10, 2014 at 10:49:51 AM, Alvaro Videla ([hidden email]) wrote:

Hi,

We have just released the RabbitMQ Sharding Plugin:

The plugin allows you to partition messages across sharded queues.
Adding new nodes to the cluster, automatically scales the shard size.

Learn more here: https://github.com/rabbitmq/rabbitmq-sharding

Any kind of feedback or questions about the plugin is welcomed.

Regards,

Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

--

Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Alvaro Videla-2]

Hi,

On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <[hidden email]> wrote:
>   The one thing I didn't see in the post was how many consumers you'd need.
> It sounds like you'd need at least N consumers where N is the number of
> nodes in the shard?  With the shoveling config, we'd need to dynamically
> grow consumers to adapt to the cluster size it sounds like.
>

Yes, at the moment you would need 1 consumer per shard. So if the
shard-per-node value is 4, then you'd need 4 consumers per node in the
cluster.

Regards,

Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[mc717990]

Since I just saw this crop up again on the list, thought I'd ask again on this plugin as it looks danged handy.  With geographically distributed data centers, we're using RabbitMQ to replicate data and information to remote data centers.  But with the WAN latency, we hit an issue where a single queue would seemingly get backlogged trying to shovel messages across the WAN.  SO we used x-consistent-hashes to basically do something like this:

Publish to a fanout exchange "bob"

bob is bound to "bob.multi"

bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through "bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system that has a single exchange "bob" with a single queue "bobs.data".  We have the shovel config over ride the routing key and exchange on the publish side so things land in one spot.  If I didn't have to script this logic out I'd be more than happy to switch to using the "shard" exchange.  And the goal on the remote side is to have a single consumer (java process) connected to queue "bobs.data" on the remote side, and a single publisher publishing to "bob" on the publish side.  I THINK I may have misunderstood the earlier comment where we'd need 1 consumer per shard thing.

Jason

On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <[hidden email]> wrote:

Hi,

On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <[hidden email]> wrote:
>   The one thing I didn't see in the post was how many consumers you'd need.
> It sounds like you'd need at least N consumers where N is the number of
> nodes in the shard?  With the shoveling config, we'd need to dynamically
> grow consumers to adapt to the cluster size it sounds like.
>

Yes, at the moment you would need 1 consumer per shard. So if the
shard-per-node value is 4, then you'd need 4 consumers per node in the
cluster.

Regards,

Alvaro

--

Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Alvaro Videla-2]

If you have 8 queues per shard, then each shard will require 8 consumers to get the data out of those 8 queues that lay behind the shard. The plugin will take care of selecting the queue with the least amount of consumers whenever a new basic.consume arrives.

On Thu, Jul 10, 2014 at 5:00 PM, Jason McIntosh <[hidden email]> wrote:

Since I just saw this crop up again on the list, thought I'd ask again on this plugin as it looks danged handy.  With geographically distributed data centers, we're using RabbitMQ to replicate data and information to remote data centers.  But with the WAN latency, we hit an issue where a single queue would seemingly get backlogged trying to shovel messages across the WAN.  SO we used x-consistent-hashes to basically do something like this:

Publish to a fanout exchange "bob"

bob is bound to "bob.multi"

bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through "bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system that has a single exchange "bob" with a single queue "bobs.data".  We have the shovel config over ride the routing key and exchange on the publish side so things land in one spot.  If I didn't have to script this logic out I'd be more than happy to switch to using the "shard" exchange.  And the goal on the remote side is to have a single consumer (java process) connected to queue "bobs.data" on the remote side, and a single publisher publishing to "bob" on the publish side.  I THINK I may have misunderstood the earlier comment where we'd need 1 consumer per shard thing.

Jason

On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <[hidden email]> wrote:

Hi,

On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <[hidden email]> wrote:
>   The one thing I didn't see in the post was how many consumers you'd need.
> It sounds like you'd need at least N consumers where N is the number of
> nodes in the shard?  With the shoveling config, we'd need to dynamically
> grow consumers to adapt to the cluster size it sounds like.
>

Yes, at the moment you would need 1 consumer per shard. So if the
shard-per-node value is 4, then you'd need 4 consumers per node in the
cluster.

Regards,

Alvaro

--

Jason McIntosh
https://github.com/jasonmcintosh/
<a href="tel:[phone]" value="[phone]" target="_blank">573-424-7612

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[mc717990]

Yeah this is still the issue for us - how would you dynamically increase your consumers easily from say a Java process?  Generally, using things like SimpleMessageListenerContainer - it starts up and consumes from a queue "foo.data" - so would we need to start up 8 separate consumer processes?  Or would a single bound consumer to that queue handle this when set to 8 concurrent consumers?  Or would we need to listen for a shard change event to allow it to do this?  I'd also ask about the shovel process - would the shovel just consume from "foo.data" and how would that handle the multiple queues?

Jason

On Thu, Jul 10, 2014 at 10:41 AM, Alvaro Videla <[hidden email]> wrote:

If you have 8 queues per shard, then each shard will require 8 consumers to get the data out of those 8 queues that lay behind the shard. The plugin will take care of selecting the queue with the least amount of consumers whenever a new basic.consume arrives.

On Thu, Jul 10, 2014 at 5:00 PM, Jason McIntosh <[hidden email]> wrote:

Since I just saw this crop up again on the list, thought I'd ask again on this plugin as it looks danged handy.  With geographically distributed data centers, we're using RabbitMQ to replicate data and information to remote data centers.  But with the WAN latency, we hit an issue where a single queue would seemingly get backlogged trying to shovel messages across the WAN.  SO we used x-consistent-hashes to basically do something like this:

Publish to a fanout exchange "bob"

bob is bound to "bob.multi"

bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through "bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system that has a single exchange "bob" with a single queue "bobs.data".  We have the shovel config over ride the routing key and exchange on the publish side so things land in one spot.  If I didn't have to script this logic out I'd be more than happy to switch to using the "shard" exchange.  And the goal on the remote side is to have a single consumer (java process) connected to queue "bobs.data" on the remote side, and a single publisher publishing to "bob" on the publish side.  I THINK I may have misunderstood the earlier comment where we'd need 1 consumer per shard thing.

Jason

On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <[hidden email]> wrote:

Hi,

On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <[hidden email]> wrote:
>   The one thing I didn't see in the post was how many consumers you'd need.
> It sounds like you'd need at least N consumers where N is the number of
> nodes in the shard?  With the shoveling config, we'd need to dynamically
> grow consumers to adapt to the cluster size it sounds like.
>

Yes, at the moment you would need 1 consumer per shard. So if the
shard-per-node value is 4, then you'd need 4 consumers per node in the
cluster.

Regards,

Alvaro

--

Jason McIntosh
https://github.com/jasonmcintosh/
<a href="tel:[phone]" value="[phone]" target="_blank">573-424-7612

--

Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Alvaro Videla-2]

You need to issue 8 basic.consume commands, whether from the same process or from a separate one. Of course you could use the same callback for all those basic.consume.

About Shovel, I don't think it will work out of the box with this, but perhaps Simon can correct me.

On Thu, Jul 10, 2014 at 6:06 PM, Jason McIntosh <[hidden email]> wrote:

Yeah this is still the issue for us - how would you dynamically increase your consumers easily from say a Java process?  Generally, using things like SimpleMessageListenerContainer - it starts up and consumes from a queue "foo.data" - so would we need to start up 8 separate consumer processes?  Or would a single bound consumer to that queue handle this when set to 8 concurrent consumers?  Or would we need to listen for a shard change event to allow it to do this?  I'd also ask about the shovel process - would the shovel just consume from "foo.data" and how would that handle the multiple queues?

Jason

On Thu, Jul 10, 2014 at 10:41 AM, Alvaro Videla <[hidden email]> wrote:

If you have 8 queues per shard, then each shard will require 8 consumers to get the data out of those 8 queues that lay behind the shard. The plugin will take care of selecting the queue with the least amount of consumers whenever a new basic.consume arrives.

On Thu, Jul 10, 2014 at 5:00 PM, Jason McIntosh <[hidden email]> wrote:

Since I just saw this crop up again on the list, thought I'd ask again on this plugin as it looks danged handy.  With geographically distributed data centers, we're using RabbitMQ to replicate data and information to remote data centers.  But with the WAN latency, we hit an issue where a single queue would seemingly get backlogged trying to shovel messages across the WAN.  SO we used x-consistent-hashes to basically do something like this:

Publish to a fanout exchange "bob"

bob is bound to "bob.multi"

bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through "bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system that has a single exchange "bob" with a single queue "bobs.data".  We have the shovel config over ride the routing key and exchange on the publish side so things land in one spot.  If I didn't have to script this logic out I'd be more than happy to switch to using the "shard" exchange.  And the goal on the remote side is to have a single consumer (java process) connected to queue "bobs.data" on the remote side, and a single publisher publishing to "bob" on the publish side.  I THINK I may have misunderstood the earlier comment where we'd need 1 consumer per shard thing.

Jason

On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <[hidden email]> wrote:

Hi,

On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <[hidden email]> wrote:
>   The one thing I didn't see in the post was how many consumers you'd need.
> It sounds like you'd need at least N consumers where N is the number of
> nodes in the shard?  With the shoveling config, we'd need to dynamically
> grow consumers to adapt to the cluster size it sounds like.
>

Yes, at the moment you would need 1 consumer per shard. So if the
shard-per-node value is 4, then you'd need 4 consumers per node in the
cluster.

Regards,

Alvaro

--

Jason McIntosh
https://github.com/jasonmcintosh/
<a href="tel:[phone]" value="[phone]" target="_blank">573-424-7612

--

Jason McIntosh
https://github.com/jasonmcintosh/
<a href="tel:[phone]" value="[phone]" target="_blank">573-424-7612

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Simon MacMullen-2]

On 10/07/2014 9:49PM, Alvaro Videla wrote:
> About Shovel, I don't think it will work out of the box with this, but
> perhaps Simon can correct me.

If you need 8 consumers you'd have to configure 8 shovels. Not
impossible, but a bit fiddly.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss