Exclusive Producers?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Exclusive Producers?

Tarun Mistry
Hi all,

i'm new to RabbitMQ/AMQP. I am trying to solve an application level high-availability problem and belive i'm not thinking about things in the right way, I could use some advice!

Here is what I would like to achieve, I could use some help understanding the RabbitMQ way of doing it.

1) I have several message producers, only 1 of these is the primary, the remainder as all backups. 
2) The primary is defined as the process that controls a single queue and can post messages to it.
3) If the primary process fails, I want a backup to take over immediately.
4) All backups are in a race to get control of the queue (from 2)
5) Once a backup gains control, it is now the primary.

Using Rabbit speak, I feel like I need something called an "exclusive producer". 

Do I need to fudge this by introducing a second queue and having all the processes connect to this using the "exclusive consumer" pattern, and then use this to determine who can control the main queue?

Appreciate any advice

Cheers
TM




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Michael Klishin-2

On 18 Feb 2014, at 06:06, Tarun Mistry <[hidden email]> wrote:

> 1) I have several message producers, only 1 of these is the primary, the remainder as all backups.
> 2) The primary is defined as the process that controls a single queue and can post messages to it.
> 3) If the primary process fails, I want a backup to take over immediately.
> 4) All backups are in a race to get control of the queue (from 2)
> 5) Once a backup gains control, it is now the primary.
>
> Using Rabbit speak, I feel like I need something called an "exclusive producer”.

There is no such feature in RabbitMQ.

> Do I need to fudge this by introducing a second queue and having all the processes connect to this using the "exclusive consumer" pattern, and then use this to determine who can control the main queue?

This is a distributed coordination problem. I’d recommend using a separate tool (ZooKeeper, Redis, etcd)
for coordination.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Simon MacMullen-2
In reply to this post by Tarun Mistry
On 18/02/2014 2:06AM, Tarun Mistry wrote:
> Using Rabbit speak, I feel like I need something called an "exclusive
> producer".

Yes, that sounds right. Sadly there's not really such a thing as a
"producer" in the sense of a client having agreed that it can produce
messages - there are just clients that may or may not publish messages.
So there's an asymmetry with consumers there.

> Do I need to fudge this by introducing a second queue and having all the
> processes connect to this using the "exclusive consumer" pattern, and
> then use this to determine who can control the main queue?

So that's probably your best bet. I can't think of anything better,
anyway. It's a bit ugly, but should be workable.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Alvaro Videla-2
In reply to this post by Tarun Mistry
Hi,

Maybe there's an easier solution, but you could try this:

- Create a queue called "lock" (or whatever name suits you) and
publish just one message to it, which will be the token producers need
to obtain in order to publish messages.
- Start your producers and make them consume just one message from
that queue, ie basic_qos(prefetch-count=1), also tell RabbitMQ that
you will ack the messages. The first producer arriving at the queue
will grab the "token". The other producers will sit idle, waiting for
said token.
- When the producer receives the "token", then it can start publishing
to the actual queue where you want the messages.
- If that consumer crashes or is shut down, then the "token" will be
requeued and some other of your backup producers will grab it, so it
can start publishing.
- if the producer wishes to stop, it can basic_reject(requeue=true)
the "token", so another producer can grab it and start publishing.

Regards,

Alvaro

On Tue, Feb 18, 2014 at 3:06 AM, Tarun Mistry <[hidden email]> wrote:

> Hi all,
>
> i'm new to RabbitMQ/AMQP. I am trying to solve an application level
> high-availability problem and belive i'm not thinking about things in the
> right way, I could use some advice!
>
> Here is what I would like to achieve, I could use some help understanding
> the RabbitMQ way of doing it.
>
> 1) I have several message producers, only 1 of these is the primary, the
> remainder as all backups.
> 2) The primary is defined as the process that controls a single queue and
> can post messages to it.
> 3) If the primary process fails, I want a backup to take over immediately.
> 4) All backups are in a race to get control of the queue (from 2)
> 5) Once a backup gains control, it is now the primary.
>
> Using Rabbit speak, I feel like I need something called an "exclusive
> producer".
>
> Do I need to fudge this by introducing a second queue and having all the
> processes connect to this using the "exclusive consumer" pattern, and then
> use this to determine who can control the main queue?
>
> Appreciate any advice
>
> Cheers
> TM
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Simon MacMullen-2
On 18/02/2014 10:23AM, Alvaro Videla wrote:
> Hi,
>
> Maybe there's an easier solution, but you could try this:

OP's suggestion of establishing an exclusive consumer to a lock queue
sounds easier :-)

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Matthias Radestock-3
In reply to this post by Simon MacMullen-2
On 18/02/14 10:18, Simon MacMullen wrote:
> On 18/02/2014 2:06AM, Tarun Mistry wrote:
>> Using Rabbit speak, I feel like I need something called an "exclusive
>> producer".
>
> Yes, that sounds right. Sadly there's not really such a thing as a
> "producer" in the sense of a client having agreed that it can produce
> messages - there are just clients that may or may not publish messages.
> So there's an asymmetry with consumers there.

Moreover, producers are not associated with single queues - messages get
published to exchanges and may end up in zero or more queues.

>> Do I need to fudge this by introducing a second queue and having all the
>> processes connect to this using the "exclusive consumer" pattern, and
>> then use this to determine who can control the main queue?
>
> So that's probably your best bet. I can't think of anything better,
> anyway. It's a bit ugly, but should be workable.

There *is* a better way. See Alvaro's reply.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Matthias Radestock-3
In reply to this post by Simon MacMullen-2
On 18/02/14 10:27, Simon MacMullen wrote:
> On 18/02/2014 10:23AM, Alvaro Videla wrote:
>> Hi,
>>
>> Maybe there's an easier solution, but you could try this:
>
> OP's suggestion of establishing an exclusive consumer to a lock queue
> sounds easier :-)

No, because it requires all other producers to poll, i.e. keep
attempting to create an exclusive consumer.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Gordon Sim-2
In reply to this post by Simon MacMullen-2
On 02/18/2014 10:18 AM, Simon MacMullen wrote:

> On 18/02/2014 2:06AM, Tarun Mistry wrote:
>> Using Rabbit speak, I feel like I need something called an "exclusive
>> producer".
>
> Yes, that sounds right. Sadly there's not really such a thing as a
> "producer" in the sense of a client having agreed that it can produce
> messages - there are just clients that may or may not publish messages.
> So there's an asymmetry with consumers there.
>
>> Do I need to fudge this by introducing a second queue and having all the
>> processes connect to this using the "exclusive consumer" pattern, and
>> then use this to determine who can control the main queue?
>
> So that's probably your best bet. I can't think of anything better,
> anyway. It's a bit ugly, but should be workable.

What I often do in cases like that is have the second queue as you
suggest, but instead of using an 'exclusive consumer' lock on it, just
have a single message in it. The consumer compete to get that one
message, when its delivered it indicates that the receiver has control.
The receiver doesn't acknowledge the message however. That means if/when
the current controller dies, the message will be automatically requeued
and redelivered to one of the backups. (Saves the backups having to
continually poll for ability to consume).

(And just in passing, I'll note that AMQP 1.0 did introduce the concept
of 'producer' of incoming messages - i.e. a sending link - analogous to
that of consumer of outgoing messages - i.e. a receiving link).

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Simon MacMullen-2
In reply to this post by Matthias Radestock-3
On 18/02/2014 10:28AM, Matthias Radestock wrote:
> No, because it requires all other producers to poll, i.e. keep
> attempting to create an exclusive consumer.

Ah, good point.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Michael Klishin-2
In reply to this post by Gordon Sim-2
On 18 Feb 2014, at 14:31, Gordon Sim <[hidden email]> wrote:

> What I often do in cases like that is have the second queue as you suggest, but instead of using an 'exclusive consumer' lock on it, just have a single message in it.

It’s worth mentioning that it is possible to enforce the single message limit:
http://www.rabbitmq.com/maxlength.html

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Alvaro Videla-2
Hi,

On Tue, Feb 18, 2014 at 11:37 AM, Michael Klishin
<[hidden email]> wrote:
> On 18 Feb 2014, at 14:31, Gordon Sim <[hidden email]> wrote:
>
>> What I often do in cases like that is have the second queue as you suggest, but instead of using an 'exclusive consumer' lock on it, just have a single message in it.
>
> It's worth mentioning that it is possible to enforce the single message limit:
> http://www.rabbitmq.com/maxlength.html

I'm not sure this will help, because "Queue length is a measure that
takes into account ready messages, ignoring unacknowledged messages
and message size."

So unacked messages, which is what this technique is issuing, won't
prevent someone from publishing a new message, thus allowing a second
producer to grab that message and start producing when it shouldn't.

Regards,

Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

eric.arendt
In reply to this post by Alvaro Videla-2
You could also use the exclusive queue consuming functionality instead of having to maintain a token message.  Make sure you're the exclusive consumer of a queue before you connect as a producer.


On Tue, Feb 18, 2014 at 2:23 AM, Alvaro Videla <[hidden email]> wrote:
Hi,

Maybe there's an easier solution, but you could try this:

- Create a queue called "lock" (or whatever name suits you) and
publish just one message to it, which will be the token producers need
to obtain in order to publish messages.
- Start your producers and make them consume just one message from
that queue, ie basic_qos(prefetch-count=1), also tell RabbitMQ that
you will ack the messages. The first producer arriving at the queue
will grab the "token". The other producers will sit idle, waiting for
said token.
- When the producer receives the "token", then it can start publishing
to the actual queue where you want the messages.
- If that consumer crashes or is shut down, then the "token" will be
requeued and some other of your backup producers will grab it, so it
can start publishing.
- if the producer wishes to stop, it can basic_reject(requeue=true)
the "token", so another producer can grab it and start publishing.

Regards,

Alvaro

On Tue, Feb 18, 2014 at 3:06 AM, Tarun Mistry <[hidden email]> wrote:
> Hi all,
>
> i'm new to RabbitMQ/AMQP. I am trying to solve an application level
> high-availability problem and belive i'm not thinking about things in the
> right way, I could use some advice!
>
> Here is what I would like to achieve, I could use some help understanding
> the RabbitMQ way of doing it.
>
> 1) I have several message producers, only 1 of these is the primary, the
> remainder as all backups.
> 2) The primary is defined as the process that controls a single queue and
> can post messages to it.
> 3) If the primary process fails, I want a backup to take over immediately.
> 4) All backups are in a race to get control of the queue (from 2)
> 5) Once a backup gains control, it is now the primary.
>
> Using Rabbit speak, I feel like I need something called an "exclusive
> producer".
>
> Do I need to fudge this by introducing a second queue and having all the
> processes connect to this using the "exclusive consumer" pattern, and then
> use this to determine who can control the main queue?
>
> Appreciate any advice
>
> Cheers
> TM
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

--
You received this message because you are subscribed to the Google Groups "rabbitmq-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/rabbitmq-discuss.
For more options, visit https://groups.google.com/groups/opt_out.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Alvaro Videla-2
On Tue, Feb 18, 2014 at 7:52 PM, Eric Arendt <[hidden email]> wrote:
> You could also use the exclusive queue consuming functionality instead of
> having to maintain a token message.  Make sure you're the exclusive consumer
> of a queue before you connect as a producer.

As far as I understand the OP is asking for exclusive *producers* not consumers.

-Alvaro
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Lukasz M.
In reply to this post by Tarun Mistry
Hi,

@Alvaro, your solution seems to be quite interesting. I just wonder if there is a simpler one.

I have a similar problem where I need to start/stop the RabbitTemplate object on demand. For example to restart the bean in my catch block with sender.stop() and then sender.start() .

My first idea was to implement SmartLifeCycle interface from Spring and extend RabbitTemplate as follows.:

@Component
public class LifeSender extends RabbitTemplate implements SmartLifecycle {

private volatile boolean isRunning = false;

@Override
public boolean isAutoStartup() {
return true;
}

@Override
public void start() {
System.out.println("STARTED!!!");
isRunning = true;
}

@Override
public void stop() {
System.out.println("STOPPED!!!");
isRunning = false;
}



Here is the Spring AMQP context:

<!-- RabbitMQ Sender. -->
<bean id="sender" class="org.springframework.amqp.rabbit.core.Rabbit Template" abstract="true">
<constructor-arg index="0" ref="connectionFactory" />
<property name="confirmCallback" ref="premCallback" />
<property name="exchange" value="myExchange" />
</bean>


<bean id="lifeSender" class="com.ucware.ucpo.cti.core.LifeSender" parent="sender"/>


The problem is that lifeSender does not instatiate the properties and connection factory from the sender. So I created the constructor:

@Autowired
public LifeSender(ConnectionFactory connectionFactory, PremConfirmCallback premCallback) {
super();
super.setConnectionFactory(connectionFactory);
super.setExchange("cti.main");
super.setConfirmCallback(premCallback);
System.out.println(super.getConnectionFactory().to String());
}


Unfortunately, lifeSender.stop() does not do more than just printing "STOPPED" to the log. I am still able to send messages.

Second idea was to use controlChannel from Spring Integration. However, a simple test

@Test
public void testTurnOffSender() throws InterruptedException {
isBean = controlGateway.isRunning( "@sender.isRunning()" );

Assert.isTrue( isBean);
controlGateway.send(new GenericMessage<String>( "@sender.stop()" ));
Thread.sleep(500);
isBean = controlGateway.isRunning( "@sender.isRunning()" );
Assert.isTrue( isBean == false);

}


triggers the exception:

org.springframework.expression.EvaluationException : The method 'isRunning' is not supported by this command processor. If using the Control Bus, consider adding @ManagedOperation or @ManagedAttribute.

Do you think any of suggested solutions make sense?

Thanks,
Lukasz


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Gary Russell-2

You need to add isRunning() as a @ManagedOperation if you want to access it using the control bus.

You would also need to override the doSend*() methods on your subclass if you want to prevent sends on a "stopped" template.


On Mon, Feb 24, 2014 at 9:10 AM, Lukasz M. <[hidden email]> wrote:
Hi,

@Alvaro, your solution seems to be quite interesting. I just wonder if there is a simpler one.

I have a similar problem where I need to start/stop the RabbitTemplate object on demand. For example to restart the bean in my catch block with sender.stop() and then sender.start() .

My first idea was to implement SmartLifeCycle interface from Spring and extend RabbitTemplate as follows.:

@Component
public class LifeSender extends RabbitTemplate implements SmartLifecycle {

private volatile boolean isRunning = false;

@Override
public boolean isAutoStartup() {
return true;
}

@Override
public void start() {
System.out.println("STARTED!!!");
isRunning = true;
}

@Override
public void stop() {
System.out.println("STOPPED!!!");
isRunning = false;
}



Here is the Spring AMQP context:

<!-- RabbitMQ Sender. -->
<bean id="sender" class="org.springframework.amqp.rabbit.core.Rabbit Template" abstract="true">
<constructor-arg index="0" ref="connectionFactory" />
<property name="confirmCallback" ref="premCallback" />
<property name="exchange" value="myExchange" />
</bean>


<bean id="lifeSender" class="com.ucware.ucpo.cti.core.LifeSender" parent="sender"/>


The problem is that lifeSender does not instatiate the properties and connection factory from the sender. So I created the constructor:

@Autowired
public LifeSender(ConnectionFactory connectionFactory, PremConfirmCallback premCallback) {
super();
super.setConnectionFactory(connectionFactory);
super.setExchange("cti.main");
super.setConfirmCallback(premCallback);
System.out.println(super.getConnectionFactory().to String());
}


Unfortunately, lifeSender.stop() does not do more than just printing "STOPPED" to the log. I am still able to send messages.

Second idea was to use controlChannel from Spring Integration. However, a simple test

@Test
public void testTurnOffSender() throws InterruptedException {
isBean = controlGateway.isRunning( "@sender.isRunning()" );

Assert.isTrue( isBean);
controlGateway.send(new GenericMessage<String>( "@sender.stop()" ));
Thread.sleep(500);
isBean = controlGateway.isRunning( "@sender.isRunning()" );
Assert.isTrue( isBean == false);

}


triggers the exception:

org.springframework.expression.EvaluationException : The method 'isRunning' is not supported by this command processor. If using the Control Bus, consider adding @ManagedOperation or @ManagedAttribute.

Do you think any of suggested solutions make sense?

Thanks,
Lukasz


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exclusive Producers?

Gary Russell-2
In reply to this post by Lukasz M.
As I mentioned on the Spring Forum...


...you need to add isRunning() as a @ManagedOperation if you want to access it using the control bus.

You would also need to override the doSend*() methods on your subclass if you want to prevent sends on a "stopped" template.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss