Implement guaranteed delivery with ReturnListener and ConfirmListener

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Implement guaranteed delivery with ReturnListener and ConfirmListener

ArnavJoshi
Hi,

We are trying to prototype guaranteed delivery of messages. RabbitMQ with basic publishing or mandatory\immediate flags only guarantees routing a message up to the broker with some asynchronous acknowledgements back to the publisher. It does not promise a reliable delivery of messages. With consumer crashes and broker restarts, the messages can be lost specifically if it is not declared as persistent.
 
We planned to implement both the ReturnListener and ConfirmListener interfaces.

a. To be notified of failed deliveries when basicPublish is called with “mandatory” flag
b. To get publisher confirms on whether messages have been consumed successfully (acks), or whether they are lost/not persisted (nack).

For testing purposes, we are checking for the confirms for messages which are being written to undeclared/unbound/currently unavailable queues. We observe that although the ReturnListener method throws a 312 (NO_ROUTE) error code, the ConfirmListener still calls the ‘handleAck’ method, and not the ‘handleNack’ method as we would expect.

If this implementation is by design – could you suggest a way in which we can implement guaranteed delivery, i.e.:

1. Confirm that the message has not only reached the broker, but has been processed by one or more queues.
2. Throw a nack, or generate an exception when the message cannot be routed and that the message has been persisted to disk
3. Making blocking calls to our publishes in the ReturnListener implementation when we are trying to write to such queues?

Follow up question: If a queue process being killed is the only criterion for ConfirmListener.handleNack being called – shouldn’t an undeclared/currently unavailable queue result in handleNack being called.
Our suspicion is that the message which cannot be routed to a currently unavailable queue is handled like an unroutable message, which is confirmed immediately after basic.return but is also ‘ACK’ed, which seems counter-intuitive.

Thanks,
Arnav Joshi
Reply | Threaded
Open this post in threaded view
|

Re: Implement guaranteed delivery with ReturnListener and ConfirmListener

Michael Klishin-2


On 14 July 2014 at 22:02:52, ArnavJoshi ([hidden email]) wrote:
> > For testing purposes, we are checking for the confirms for messages  
> which
> are being written to undeclared/unbound/currently unavailable  
> queues. We
> observe that although the ReturnListener method throws a 312  
> (NO_ROUTE)
> error code, the ConfirmListener still calls the ‘handleAck’  
> method, and not
> the ‘handleNack’ method as we would expect.

I can see that some may expect unroutable messages
to be nack-ed but RabbitMQ cannot know whether that's the case: in some cases
it's fine for messages to not be routed anywhere. So acks verify a bunch of things:

 1. Your connection is alive
 2. Your channel is still open (no exceptions)
 3. Routed message was delivered to all queue mirrors (if applicable)

Using a nack to indicate that the message is unroutable would confuse a lot
of apps into thinking that 1) or 2) were not true.


> If this implementation is by design – could you suggest a way in  
> which we
> can implement guaranteed delivery, i.e.:
>  
> 1. Confirm that the message has not only reached the broker, but  
> has been
> processed by one or more queues.
> 2. Throw a nack, or generate an exception when the message cannot  
> be routed
> and that the message has been persisted to disk
> 3. Making blocking calls to our publishes in the ReturnListener  
> implementation when we are trying to write to such queues?

Group messages in batches if you can. Then
publish a batch, use Channel#waitForConfirms. If it returns false, schedule
the batch for re-publishing. If a return arrives, do the same.
You'll also need to have a way to identify messages, but this is generally true
with confirms.

Batching will provide good throughput with easier to use but generally slower API (#waitForConfirms vs. ConfirmListener). 

Because basic.return may arrive after #waitForConfirms returns, you may want to schedule
a "batch check" that runs in a few seconds and completely marks the batch as successfully
delivered.

> Follow up question: If a queue process being killed is the only  
> criterion
> for ConfirmListener.handleNack being called – shouldn’t an  
> undeclared/currently unavailable queue result in handleNack  
> being called.
> Our suspicion is that the message which cannot be routed to a currently  
> unavailable queue is handled like an unroutable message, which  
> is confirmed
> immediately after basic.return but is also ‘ACK’ed, which seems  
> counter-intuitive.

Can you expand on how exactly your experiment was conducted? Did you receive
an ack for a message routed to a queue whose master was unavailable? Was it the only
queue the message was routed to? 
-- 
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Implement guaranteed delivery with ReturnListener and ConfirmListener

ArnavJoshi
> Did you receive an ack for a message routed to a queue whose master was unavailable?

We experimented by sending messages to undeclared exchanges and undeclared queues. We received an individual ack for every message that was not routed to the exchange and a specific queue.

> Was it the only queue the message was routed to?

Yes. We specify separate routing keys for 2 queues. There is no overlap.

Our architecture needs acknowledgements for individual messages, especially when they are flagged as mandatory and need to be persisted to disk in cases of failure. Hence, grouping messages in batches and then performing a "batch check" is not what we are looking out for.

Can you provide an example where the producer receives an ack on whether a message was persisted to disk, in case when an exchange or a queue are undeclared/currently unavailable?
Reply | Threaded
Open this post in threaded view
|

Re: Implement guaranteed delivery with ReturnListener and ConfirmListener

Michael Klishin-2
On 17 July 2014 at 01:37:23, ArnavJoshi ([hidden email]) wrote:
> > We experimented by sending messages to undeclared exchanges
> and undeclared
> queues. We received an individual ack for every message that
> was not routed
> to the exchange and a specific queue.

Publishing to an undeclared exchange should result in a channel error (404 NOT_FOUND).

D id you use exchange-to-exchange bindings?
--
MK

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss