High Scalability and High Availability...How to?!!

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

High Scalability and High Availability...How to?!!

Shadi Mari
Hello,

I am new to Rabbitmq, however i have built so many messaging solution on top of other stacks..

I am trying to leverage the capabilities of RabbitMQ in one of my solutions where HA and high scalability  is a must; so far the better option the came out of my research 
is to use active/active cluster and mirrored queues in order to provide HA, otherwise using active/standby where the queue lives on the node it was created on, will not achieve HA requirement.

On the other hand, this should be mixed a load balancer, where consumers and publishers are connected to the cluster through in order to direct consumers/publishers to a different node each time the connection takes place.

However, my understanding says that RabbiqMQ in the case of mirrored queue will connect consumers and publishers to the master node which will not help in distributing load across all nodes. I am not sure but it came to my knowledge that SoundCloud uses the same approach!

Is my understanding above correct? if Yes, then how do companies that uses RabbitMQ and require HA and high scalability work it out.

Thank you

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Michael Klishin-2
 On 12 June 2014 at 21:52:47, Shadi Mari ([hidden email]) wrote:
> > However, my understanding says that RabbiqMQ in the case of  
> mirrored queue will connect consumers and publishers to the  
> master node which will not help in distributing load across all  
> nodes

"master" in the above does not mean a single cluster-wide master node. It means "master
for a particular queue". By spreading connections between nodes using a balancer,
you (should) spread the nodes where queues were declared (using queue.declare).
The node on which a queue was declared becomes its master.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Shadi Mari
Does that mean that if  i have two publishers declare the same queue on different nodes of the cluster, i will end up having two master queues and two slaves?

What is the right way and best practice to mix load balancing with HA on RabbitMQ?

Shadi


On Thu, Jun 12, 2014 at 9:02 PM, Michael Klishin <[hidden email]> wrote:
 On 12 June 2014 at 21:52:47, Shadi Mari ([hidden email]) wrote:
> > However, my understanding says that RabbiqMQ in the case of
> mirrored queue will connect consumers and publishers to the
> master node which will not help in distributing load across all
> nodes

"master" in the above does not mean a single cluster-wide master node. It means "master
for a particular queue". By spreading connections between nodes using a balancer,
you (should) spread the nodes where queues were declared (using queue.declare).
The node on which a queue was declared becomes its master.
--
MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Michael Klishin-2
 On 12 June 2014 at 22:18:27, Shadi Mari ([hidden email]) wrote:
> > Does that mean that if i have two publishers declare the same  
> queue on different nodes of the cluster, i will end up having two  
> master queues and two slaves?

queue.declare is idempotent. If you declare a queue that already exists
with the same attributes, nothing happens. If the attributes differ, you get
an error.

> What is the right way and best practice to mix load balancing with  
> HA on RabbitMQ?

"the right way" depends on what you do. Having a load balancer in front of
a cluster to distribute queue masters is one technique. Not mirroring to all
nodes in the cluster is another (2-3 nodes is often sufficient). Having
your consumers connect to queue masters to avoid intra-cluster traffic
(you can infer queue masters from queue info over HTTP API) will 
increase throughput.

The goal is to spread queue masters between nodes and improve data locality.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Paul Holyoake
In reply to this post by Shadi Mari
I've recently had a similar HA and scalability requirement. I couldn't find a perfect answer. I achieved HA through a network loadbalancer across a pair of clustered RabbitMQ nodes, with mirrored queues. If the NLB fails to connect on port 5672 of the primary it failsover to the secondary node. All queue data is available on this node due to the queue mirroring.

In order to achieve scalability my assumption is to logically partition the data messages and have over clusters to meet that requirement.

Would the NLB in round robin mode be viable and have the nodes effectively active/active?



On Thursday, June 12, 2014 5:47:30 PM UTC+1, Shadi Mari wrote:
Hello,

I am new to Rabbitmq, however i have built so many messaging solution on top of other stacks..

I am trying to leverage the capabilities of RabbitMQ in one of my solutions where HA and high scalability  is a must; so far the better option the came out of my research 
is to use active/active cluster and mirrored queues in order to provide HA, otherwise using active/standby where the queue lives on the node it was created on, will not achieve HA requirement.

On the other hand, this should be mixed a load balancer, where consumers and publishers are connected to the cluster through in order to direct consumers/publishers to a different node each time the connection takes place.

However, my understanding says that RabbiqMQ in the case of mirrored queue will connect consumers and publishers to the master node which will not help in distributing load across all nodes. I am not sure but it came to my knowledge that SoundCloud uses the same approach!

Is my understanding above correct? if Yes, then how do companies that uses RabbitMQ and require HA and high scalability work it out.

Thank you

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

mc717990
Paul:

You'll get load balanced connections this way to the cluster.  But queues are 'mastered' on a single node in the cluster, and mirrored to other nodes.  You can control somewhat which node the queues are mastered on.  If you're declaring the queues in your publisher/consumer, then the queues are mastered I believe on the connected node, so load balancing would get you there.  For mirroring though, you can say build a 5 node cluster, mirror your queue to 3 nodes, and then try and spread your queues out across the nodes in such a way that allows growth.  You have redundancy x3, and the ability to grow performance wise.  The biggest trick here is that your apps may connect to a node that the queue isn't mastered on but it seems the rabbit server (assuming it's not a master) acts as a proxy to the mastered queue.  And you get pretty solid scalability/performance with such an architecture.  At least, we have :)

Jason


On Fri, Jun 13, 2014 at 2:40 AM, Paul Holyoake <[hidden email]> wrote:
I've recently had a similar HA and scalability requirement. I couldn't find a perfect answer. I achieved HA through a network loadbalancer across a pair of clustered RabbitMQ nodes, with mirrored queues. If the NLB fails to connect on port 5672 of the primary it failsover to the secondary node. All queue data is available on this node due to the queue mirroring.

In order to achieve scalability my assumption is to logically partition the data messages and have over clusters to meet that requirement.

Would the NLB in round robin mode be viable and have the nodes effectively active/active?



On Thursday, June 12, 2014 5:47:30 PM UTC+1, Shadi Mari wrote:
Hello,

I am new to Rabbitmq, however i have built so many messaging solution on top of other stacks..

I am trying to leverage the capabilities of RabbitMQ in one of my solutions where HA and high scalability  is a must; so far the better option the came out of my research 
is to use active/active cluster and mirrored queues in order to provide HA, otherwise using active/standby where the queue lives on the node it was created on, will not achieve HA requirement.

On the other hand, this should be mixed a load balancer, where consumers and publishers are connected to the cluster through in order to direct consumers/publishers to a different node each time the connection takes place.

However, my understanding says that RabbiqMQ in the case of mirrored queue will connect consumers and publishers to the master node which will not help in distributing load across all nodes. I am not sure but it came to my knowledge that SoundCloud uses the same approach!

Is my understanding above correct? if Yes, then how do companies that uses RabbitMQ and require HA and high scalability work it out.

Thank you

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




--
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Shadi Mari
In reply to this post by Michael Klishin-2
Thanks for clarifying this Michael. I am trying to reflect what you have said on my scenario...

Let's say i have two nodes N1 and N2, and one Queue that is mirrored, the master and slave reside on N1 and N2 respectively; in such a case and as based on what you have advised, there is no way to distribute consumers and publishers load across two nodes because the same work will be done by the two nodes; thus to rectify, load distribution manifests itself when we have two or more mirrored queues distributes across 3 or more machines...

Is my assumption correct?

Thank you


On Thu, Jun 12, 2014 at 9:21 PM, Michael Klishin <[hidden email]> wrote:
 On 12 June 2014 at 22:18:27, Shadi Mari ([hidden email]) wrote:
> > Does that mean that if i have two publishers declare the same
> queue on different nodes of the cluster, i will end up having two
> master queues and two slaves?

queue.declare is idempotent. If you declare a queue that already exists
with the same attributes, nothing happens. If the attributes differ, you get
an error.

> What is the right way and best practice to mix load balancing with
> HA on RabbitMQ?

"the right way" depends on what you do. Having a load balancer in front of
a cluster to distribute queue masters is one technique. Not mirroring to all
nodes in the cluster is another (2-3 nodes is often sufficient). Having
your consumers connect to queue masters to avoid intra-cluster traffic
(you can infer queue masters from queue info over HTTP API) will 
increase throughput.

The goal is to spread queue masters between nodes and improve data locality.
--
MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Michael Klishin-2

 On 15 June 2014 at 12:13:28, Shadi Mari ([hidden email]) wrote:
> > Let's say i have two nodes N1 and N2, and one Queue that is mirrored,  
> the master and slave reside on N1 and N2 respectively; in such  
> a case and as based on what you have advised, there is no way to distribute  
> consumers and publishers load across two nodes because the same  
> work will be done by the two nodes; thus to rectify, load distribution  
> manifests itself when we have two or more mirrored queues distributes  
> across 3 or more machines...
>  
> Is my assumption correct?

It is. A single giant queue is problematic in other ways, too, so avoid this
scenario.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Shadi Mari
Sounds great!

One more thing i was not lucky enough to find is to explore various performance benchmarks were done against several rabbitmq deployments that will me to size a rabbitmq solution.

For instance what would require to achieve 15K+ msgs/second? given that i have multiple publishers and consumers on the same queue and the queue content is draining (i.e.. no accumulation)? do i need to have an SSD to increase throughput or SATA will be enough?

Thanks


On Sun, Jun 15, 2014 at 3:56 PM, Michael Klishin <[hidden email]> wrote:

 On 15 June 2014 at 12:13:28, Shadi Mari ([hidden email]) wrote:
> > Let's say i have two nodes N1 and N2, and one Queue that is mirrored,
> the master and slave reside on N1 and N2 respectively; in such
> a case and as based on what you have advised, there is no way to distribute
> consumers and publishers load across two nodes because the same
> work will be done by the two nodes; thus to rectify, load distribution
> manifests itself when we have two or more mirrored queues distributes
> across 3 or more machines...
>
> Is my assumption correct?

It is. A single giant queue is problematic in other ways, too, so avoid this
scenario.
--
MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: High Scalability and High Availability...How to?!!

Michael Klishin-2
On 15 June 2014 at 17:15:48, Shadi Mari ([hidden email]) wrote:
> > For instance what would require to achieve 15K+ msgs/second?  
> given that i have multiple publishers and consumers on the same  
> queue and the queue content is draining (i.e.. no accumulation)?  
> do i need to have an SSD to increase throughput or SATA will be enough?  

It depends on message size and if messages are published as persistent
but with commonly small messages (<= 1Kb), 15K/second
can be achieved with a single node, a single queue, and  modest hardware.

You likely won't need an SSD, and SSD won't make any difference if you publish
messages as transient.

But don't take my word for it, run your own benchmarks. There is a tool
for that: http://www.rabbitmq.com/java-tools.html

(and a couple of alternatives on GitHub, too)
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss