Push to back of Queue on NAck

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Push to back of Queue on NAck

Will Hughes

This is following on from a tweet I made earlier about wishing that RabbitMQ had the ability to push messages to the back of the queue when NAcked. 
Alvaro Videla replied suggesting I ask for the feature here. 


Scenario: 
I'm processing messages from a queue with confirmation required. 
The number of messages relative to the consumption rate can be quite high at times. In excess of hundreds of thousands of messages, with consumption rate in the low hundreds per second range. 

Sometimes we have to abandon processing a message for some transient reason (eg: external resource is locked/unavailable), and so we send a NAck.  
At the moment, RabbitMQ will immediately redeliver that message to any of the consumers. If the transient condition is still in place, we can potentially get into a tight loop where we're constantly consuming the same message(s) hundreds of times per second. 

For our scenario, we *can* duplicate the message, send it to the exchange/queue, and Ack the original. 

Something I thought might be a bit nicer is if we could have an option to have RabbitMQ put the message at the back of the queue. Depending on the size of the queue and our processing rate, this might give us an easier way to try other messages in the queue first. 

Another option I was thinking of to enforce more a delay was to see if I could chain together two queues using dead-letter-exchange settings: 

Queue A, no TTL, DLX=Exchange B (with Queue B bound to it). 
Queue B, ttl of (x) seconds, DLX= Exchange A (with Queue A bound to it). 

When we NAck the mesage from Queue A, we set requeue=false. 

I've not seen much discussion so far on whether it's sane to hook up two queues/exchanges in a loop like this. 
Closest I've seen is this: http://yuserinterface.com/dev/2013/01/08/how-to-schedule-delay-messages-with-rabbitmq-using-a-dead-letter-exchange/ but that's just using TTL to DLX to delay processing intially, not the whole chain. 

Suggestions or comments? 

Cheers
Will Hughes. 

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Matthias Radestock-3
On 26/07/13 13:25, Will Hughes wrote:

> Scenario:
> I'm processing messages from a queue with confirmation required.
> The number of messages relative to the consumption rate can be quite
> high at times. In excess of hundreds of thousands of messages, with
> consumption rate in the low hundreds per second range.
>
> Sometimes we have to abandon processing a message for some transient
> reason (eg: external resource is locked/unavailable), and so we send a
> NAck.
> At the moment, RabbitMQ will immediately redeliver that message to any
> of the consumers. If the transient condition is still in place, we can
> potentially get into a tight loop where we're constantly consuming the
> same message(s) hundreds of times per second.
>
> For our scenario, we *can* duplicate the message, send it to the
> exchange/queue, and Ack the original.

The problem here is that with a short queue you still end up in a tight
loop.

> Something I thought might be a bit nicer is if we could have an option
> to have RabbitMQ put the message at the back of the queue. Depending on
> the size of the queue and our processing rate, this might give us an
> easier way to try other messages in the queue first.

This suffers from the same problem.

> Another option I was thinking of to enforce more a delay was to see if I
> could chain together two queues using dead-letter-exchange settings:
>
> Queue A, no TTL, DLX=Exchange B (with Queue B bound to it).
> Queue B, ttl of (x) seconds, DLX= Exchange A (with Queue A bound to it).
>
> When we NAck the mesage from Queue A, we set requeue=false.
>
> I've not seen much discussion so far on whether it's sane to hook up two
> queues/exchanges in a loop like this.

Yep, that makes perfect sense and should work well.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Tom Anderson
On 26/07/13 14:18, Matthias Radestock wrote:
On 26/07/13 13:25, Will Hughes wrote:

Another option I was thinking of to enforce more a delay was to see if I
could chain together two queues using dead-letter-exchange settings:

Queue A, no TTL, DLX=Exchange B (with Queue B bound to it).
Queue B, ttl of (x) seconds, DLX= Exchange A (with Queue A bound to it).

When we NAck the mesage from Queue A, we set requeue=false.

I've not seen much discussion so far on whether it's sane to hook up two
queues/exchanges in a loop like this.

Yep, that makes perfect sense and should work well.

Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

If you want exponential backoff in the retries, then life gets more complicated (multiple timeout queues, selected between by a routing key set by the consumer of A?). We are currently pussyfooting around this issue at my company. I will report back here if we ever implement a good solution!

tom

--

Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com

STATEMENT OF CONFIDENTIALITY: The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify Tom Anderson at TIM Group at [hidden email] and destroy all copies of this message and any attachments.

TIM Group is the trading name for YouDevise Limited. YouDevise Limited is registered in England, No. 3331176. Registered office: 3 Copthall Avenue, London, EC2R 7BH.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Ceri Storey
(26/07/13 16:49), Tom Anderson wrote:
Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

If you want exponential backoff in the retries, then life gets more complicated (multiple timeout queues, selected between by a routing key set by the consumer of A?). We are currently pussyfooting around this issue at my company. I will report back here if we ever implement a good solution!

I've just written some code to do exactly this; limited retries with exponential backoff. That said, we're kind of cheating in that we store the retry state in a secondary datastore and buffer messages in the client.

So whenever we receive a message, we:
  • When we post each message, we assign it a unique message_id
  • Lookup message's due time by it's message_id property in our datastore
  • Stash the message in a heap queue
  • When the message becomes due, remove it from the heap queue and pass it to the client code.
  • If the client code succeeds, then we finally ack the message. Otherwise, we reject the message.

Whilst you can scale this horizontally, you will need enough buffer space to hold a reasonable proportion of your queue, although what proportion depends on how much you care about timeliness. Also, you are effectively implementing a secondary queue in your application (retaining rabbit for it's reliability properties), which seems less than ideal.



tom

--

Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com

STATEMENT OF CONFIDENTIALITY: The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify Tom Anderson at TIM Group at [hidden email] and destroy all copies of this message and any attachments.

TIM Group is the trading name for YouDevise Limited. YouDevise Limited is registered in England, No. 3331176. Registered office: 3 Copthall Avenue, London, EC2R 7BH.



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Tom Anderson
On 26/07/13 17:07, Ceri Storey wrote:
(26/07/13 16:49), Tom Anderson wrote:
Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

If you want exponential backoff in the retries, then life gets more complicated (multiple timeout queues, selected between by a routing key set by the consumer of A?). We are currently pussyfooting around this issue at my company. I will report back here if we ever implement a good solution!

I've just written some code to do exactly this; limited retries with exponential backoff. That said, we're kind of cheating in that we store the retry state in a secondary datastore and buffer messages in the client.

So whenever we receive a message, we:
  • When we post each message, we assign it a unique message_id
  • Lookup message's due time by it's message_id property in our datastore
  • Stash the message in a heap queue
  • When the message becomes due, remove it from the heap queue and pass it to the client code.
  • If the client code succeeds, then we finally ack the message. Otherwise, we reject the message.

I'm currently writing almost exactly the same thing! The difference being that i'm putting the due time in a header on the message rather than in a lookaside store, and that my component moves messages from a queue to an exchange, rather than from a queue to client code directly.

Whilst you can scale this horizontally, you will need enough buffer space to hold a reasonable proportion of your queue, although what proportion depends on how much you care about timeliness.


I'm not sure i understand. Don't you need to have enough space to hold all the messages that could be delayed at any given time? In our case, that happens to not be all that large, fortunately.

Also, you are effectively implementing a secondary queue in your application (retaining rabbit for it's reliability properties), which seems less than ideal.


Yes. We did kick around the idea of writing a custom exchange to do this, but that sounded really scary.

tom

--

Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com

STATEMENT OF CONFIDENTIALITY: The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify Tom Anderson at TIM Group at [hidden email] and destroy all copies of this message and any attachments.

TIM Group is the trading name for YouDevise Limited. YouDevise Limited is registered in England, No. 3331176. Registered office: 3 Copthall Avenue, London, EC2R 7BH.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Ceri Storey
(26/07/13 18:03), Tom Anderson wrote:
On 26/07/13 17:07, Ceri Storey wrote:
(26/07/13 16:49), Tom Anderson wrote:
Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

If you want exponential backoff in the retries, then life gets more complicated (multiple timeout queues, selected between by a routing key set by the consumer of A?). We are currently pussyfooting around this issue at my company. I will report back here if we ever implement a good solution!

I've just written some code to do exactly this; limited retries with exponential backoff. That said, we're kind of cheating in that we store the retry state in a secondary datastore and buffer messages in the client.

So whenever we receive a message, we:
  • When we post each message, we assign it a unique message_id
  • Lookup message's due time by it's message_id property in our datastore
  • Stash the message in a heap queue
  • When the message becomes due, remove it from the heap queue and pass it to the client code.
  • If the client code succeeds, then we finally ack the message. Otherwise, we reject the message.

I'm currently writing almost exactly the same thing! The difference being that i'm putting the due time in a header on the message rather than in a lookaside store, and that my component moves messages from a queue to an exchange, rather than from a queue to client code directly.

Whilst you can scale this horizontally, you will need enough buffer space to hold a reasonable proportion of your queue, although what proportion depends on how much you care about timeliness.


I'm not sure i understand. Don't you need to have enough space to hold all the messages that could be delayed at any given time? In our case, that happens to not be all that large, fortunately.
I've set the prefetch buffer set to a conservative number of messages; mostly to avoid accidentally causing a denial of service of my own application. To be clear, this will mean that we don't see some messages until after they become due, but that's okay in our case.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

William Hughes
On Fri, Jul 26, 2013 at 11:18 PM, Matthias Radestock <[hidden email]> wrote:
Queue A, no TTL, DLX=Exchange B (with Queue B bound to it).
Queue B, ttl of (x) seconds, DLX= Exchange A (with Queue A bound to it).

When we NAck the mesage from Queue A, we set requeue=false.

Yep, that makes perfect sense and should work well.


I'll give the using-deadletter-exchange with NAck approach a go. 



On Sat, Jul 27, 2013 at 1:49 AM, Tom Anderson <[hidden email]> wrote:

Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

 
Yep, we've got other ways to deal with permanent failures/poison messages. 
This is purely to deal with transient problems where we might need to try several times. 


Thanks for the feedback and discussion everyone. 
Cheers


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Stefan Kaes-3
Hi,

I tried this, with a slight modification:

Instead of having a dedicated second exchange, I use the default exchange and set the routing key to the name of the queue I want it routed to.

Queue A, no TTL, x-dead-letter-exchange='', x-dead-letter-routing-key='B'
Queue B, ttl of (x) seconds, x-dead-letter-exchange='', x-dead-letter-routing-key='A'
 
Works like a charm.

However, there's a problem, which I think will prevent us from using this approach:

The size of the x-death header attached to the message grows linearly with the number of dead lettering events.

This is bad for performance, and in the worst case could crash rabbitmq with an out of memory error, if for some reason a message gets rejected a high number of times (or indefinitely).

It would be great if the x-death header could be limited to have only a bounded number entries, say the last 10 events.

Plus maybe the total number of redirects.

Cheers,

Stefan Kaes



On Fri, Jul 26, 2013 at 11:18 PM, Matthias Radestock <[hidden email]> wrote:
Queue A, no TTL, DLX=Exchange B (with Queue B bound to it).
Queue B, ttl of (x) seconds, DLX= Exchange A (with Queue A bound to it).

When we NAck the mesage from Queue A, we set requeue=false.

Yep, that makes perfect sense and should work well.


I'll give the using-deadletter-exchange with NAck approach a go. 



On Sat, Jul 27, 2013 at 1:49 AM, Tom Anderson <[hidden email]> wrote:

Implemented exactly as described there, it yields an infinite loop for unprocessable messages. You might therefore also want to keep a count of the number of processing attempts in a header on the message, and more thoroughly reject messages which reach some maximum number of attempts. I think you could do the final rejection by setting a routing key on the message when you reject it for the last time, and having exchange B be a direct exchange which routes to either queue B or some final deadletter queue.

 
Yep, we've got other ways to deal with permanent failures/poison messages. 
This is purely to deal with transient problems where we might need to try several times. 


Thanks for the feedback and discussion everyone. 
Cheers


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Matthias Radestock-3
Stefan,

On 28/07/13 15:45, Stefan Kaes wrote:
> The size of the x-death header attached to the message grows linearly
> with the number of dead lettering events.
>
> This is bad for performance, and in the worst case could crash rabbitmq
> with an out of memory error, if for some reason a message gets rejected
> a high number of times (or indefinitely).

Surely the TTL-induced delay will be high enough for this not to be a
problem in practice. After all, the x-death header only takes up a few
tens of bytes per entry.

Furthermore, surely you wouldn't want a message to get rejected
indefinitely. At some point the app presumably will have to give up and
either throw away the message or route it somewhere for manual
intervention. The app can make that decision based on the length of the
x-death header.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Push to back of Queue on NAck

Brian Reischl
In reply to this post by Will Hughes
In the looped scenario, I would worry about messages cycling endlessly. eg, it hits some bug in your code, gets dead lettered, gets retried and hits the same bug, gets dead lettered, and so on. I'd want some mechanism to stop retrying the message after some number of cycles, or move it into a dead-letter queue that doesn't go anywhere (a dead-end dead-letter). 

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss