Bound Queues

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Bound Queues

Dan Di Spaltro
Is there anyway you can specify a memory/disk/length limit to a queue
size with the action being a similar to a fixed window?

If not, is there any chance it is on the roadmap?  I know its not in
the amqp spec, but it could be treated like a header.

--
Dan Di Spaltro

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Tony Garnock-Jones-2
Dan Di Spaltro wrote:
> Is there anyway you can specify a memory/disk/length limit to a queue
> size with the action being a similar to a fixed window?

Not at the moment, no.

> If not, is there any chance it is on the roadmap?  I know its not in
> the amqp spec, but it could be treated like a header.

It's not exactly on the roadmap, but it's something we've considered before.
There are some questions that need careful thought though, some of which have
more than one valid answer, mostly around the issue of what should happen once
a configured limit is reached. Would you expect newly-arriving messages to be
dropped? Would you expect the oldest enqueued messages to be dropped? Would you
expect to be warned about the dropped messages? Would you expect dropped
messages to be re-published somewhere, through some kind of dead-letter
exchange? How would queue limits interact with
delivered-but-not-yet-acknowledged messages? How would queue limits interact
with transactionally-published messages? How would queue limits interact with
the "mandatory" and "immediate" flags?

Regards,
  Tony

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Matthias Radestock-2
Dan,

Tony Garnock-Jones wrote:
> Dan Di Spaltro wrote:
>> Is there anyway you can specify a memory/disk/length limit to a queue
>> size with the action being a similar to a fixed window? [...]
>
> It's not exactly on the roadmap, but it's something we've considered before.
> There are some questions that need careful thought though [...]

One particular tricky issue is sharing. AMQP message may get delivered
to 0, 1 or *several* queues. RabbitMQ generally only keeps *one* copy of
a message, regardless of how many queues it ends up in. So queues end up
sharing messages. And some data can even be shared *between* messages.

Hence there really is no such thing as a per-queue memory/disk footprint.

Furthermore, there is no way to determine the memory footprint of a
message precisely due to memory fragmentation, gc, etc.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Jeff Lindsay
I didn't realize this about queues. I thought that's where the state was in Rabbit. For example, I thought when a node goes down, the messages in the queues in that node are lost. 

On Mon, Apr 12, 2010 at 10:25 PM, Matthias Radestock <[hidden email]> wrote:
Dan,

Tony Garnock-Jones wrote:
> Dan Di Spaltro wrote:
>> Is there anyway you can specify a memory/disk/length limit to a queue
>> size with the action being a similar to a fixed window? [...]
>
> It's not exactly on the roadmap, but it's something we've considered before.
> There are some questions that need careful thought though [...]

One particular tricky issue is sharing. AMQP message may get delivered
to 0, 1 or *several* queues. RabbitMQ generally only keeps *one* copy of
a message, regardless of how many queues it ends up in. So queues end up
sharing messages. And some data can even be shared *between* messages.

Hence there really is no such thing as a per-queue memory/disk footprint.

Furthermore, there is no way to determine the memory footprint of a
message precisely due to memory fragmentation, gc, etc.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Tony Garnock-Jones-2
Jeff Lindsay wrote:
> I didn't realize this about queues. I thought that's where the state was
> in Rabbit. For example, I thought when a node goes down, the messages in
> the queues in that node are lost.

Ah, I think Matthias is glossing over issues of clustering here and talking
just about a single node's behaviour. So where he says RabbitMQ keeps one copy
regardless of how many queues, he means each node that holds queues that a
message is destined for keeps one copy. Generally. There are lots of caveats.
The point, though, is that one message may end up on 100 queues, but be shared
between them, leading to nontrivial memory accounting.

In this case I'd be inclined to do the simplest thing: count each message on a
queue toward its budget, no matter if the system has cleverly managed to avoid
copying that same message out to a million other queues. But it really does
suggest that we're not really measuring the right thing here somehow; I'm
looking forward to discovering a better way of accounting for messages!

Tony


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Garrett Smith-5
In reply to this post by Matthias Radestock-2
On Tue, Apr 13, 2010 at 12:25 AM, Matthias Radestock
<[hidden email]> wrote:

> Dan,
>
> Tony Garnock-Jones wrote:
>> Dan Di Spaltro wrote:
>>> Is there anyway you can specify a memory/disk/length limit to a queue
>>> size with the action being a similar to a fixed window? [...]
>>
>> It's not exactly on the roadmap, but it's something we've considered before.
>> There are some questions that need careful thought though [...]
>
> One particular tricky issue is sharing. AMQP message may get delivered
> to 0, 1 or *several* queues. RabbitMQ generally only keeps *one* copy of
> a message, regardless of how many queues it ends up in. So queues end up
> sharing messages. And some data can even be shared *between* messages.
>
> Hence there really is no such thing as a per-queue memory/disk footprint.
>
> Furthermore, there is no way to determine the memory footprint of a
> message precisely due to memory fragmentation, gc, etc.

At the end of the day though, when messages pile up in a queue (just
takes one), you'll run out of memory at some point.

Without support for ttl/expires, Rabbit is particularly susceptible to
this "problem". I realize this it's by design and there are cases
where you *want* this, but there are absolutely cases when you
*don't*. It just takes one flaky consumer to put your entire system in
jeopardy.

For this reason, I've moved all of our queues to exclusive + auto
delete. I'd prefer to maintain queue state as producers and consumers
drop off, but the benefit of this is not enough to justify the
risk/cost.

I also have an external monitor on queue size (I believe measured in
memory as per rabbitmqctl output) in the event a rogue consumer
declares a queue with the wrong settings.

FWIW, the qpid server does respect the ttl message property and this
condition is easier to manage as a result. I'm a fan of ttl as it's
application specific (producers get to chose how long their messages
should live) and provides a reasonable heuristic for deleting messages
that aren't consumed in a timely manner.

This has been brought up a few times and I understand that there is a
similar set of questions that need to be addressed. Nonetheless, from
my point of view, this is a topic (general problem) that should get a
bump in road map priority.

Garrett

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Matthias Radestock-3
Garrett,

Garrett Smith wrote:
> At the end of the day though, when messages pile up in a queue (just
> takes one), you'll run out of memory at some point.

You won't run out of memory with the new persister, which has been a top
development priority for some time. Sorry it's taking so long to get
that into an official release.

But even with memory consumption no longer being an issue it may still
be desirable to drop messages at some point. Hence ...

> Without support for ttl/expires, Rabbit is particularly susceptible to
> this "problem". I realize this it's by design

ttl/expires are on the todo list. They are not missing "by design".

> from my point of view, this is a topic (general problem) that should
> get a bump in road map priority.

Ack. FWIW, you could implement a crude ttl/expires logic purely as an
AMQP client that periodically fetches all messages from a queue and acks
all those that have expired. Obviously that is far less efficient than
what could be done inside the broker. Plus it affects message order. But
it's probably good enough for a whole bunch of use cases.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Tony Garnock-Jones-2
In reply to this post by Garrett Smith-5
Garrett Smith wrote:
> FWIW, the qpid server does respect the ttl message property and this
> condition is easier to manage as a result. I'm a fan of ttl as it's
> application specific (producers get to chose how long their messages
> should live) and provides a reasonable heuristic for deleting messages
> that aren't consumed in a timely manner.

And as it happens, ttl is one of the key controls of reliable delivery: in
order to achieve reliable delivery, you need to be able to bound the lifetime
of a packet in the network, so that duplicates don't outlive their
deduplication buffer entries.

I think implementing ttl is important. We could build a dumb form of it quickly
enough, where as a message is considered for delivery its ttl is checked, but
the real challenge lies in collecting expired messages while they're idle.
Per-message ttls aggravate the problem: if only it were a per *queue* ttl, it'd
be so much easier!

Tony


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Garrett Smith-5
In reply to this post by Matthias Radestock-3
On Wed, Apr 14, 2010 at 6:00 AM, Matthias Radestock
<[hidden email]> wrote:

> Garrett,
>
> Garrett Smith wrote:
>>
>> At the end of the day though, when messages pile up in a queue (just
>> takes one), you'll run out of memory at some point.
>
> You won't run out of memory with the new persister, which has been a top
> development priority for some time. Sorry it's taking so long to get that
> into an official release.
>
> But even with memory consumption no longer being an issue it may still be
> desirable to drop messages at some point. Hence ...
>
>> Without support for ttl/expires, Rabbit is particularly susceptible to
>> this "problem". I realize this it's by design
>
> ttl/expires are on the todo list. They are not missing "by design".

Awesome!

>> from my point of view, this is a topic (general problem) that should
>> get a bump in road map priority.
>
> Ack. FWIW, you could implement a crude ttl/expires logic purely as an AMQP
> client that periodically fetches all messages from a queue and acks all
> those that have expired. Obviously that is far less efficient than what
> could be done inside the broker. Plus it affects message order. But it's
> probably good enough for a whole bunch of use cases.

And another moving part for the app, but still... good idea.

Garrett

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Dan Di Spaltro
In reply to this post by Tony Garnock-Jones-2
ttl's on a queue level would be very useful to us

On Wed, Apr 14, 2010 at 4:13 AM, Tony Garnock-Jones <[hidden email]> wrote:

> Garrett Smith wrote:
>> FWIW, the qpid server does respect the ttl message property and this
>> condition is easier to manage as a result. I'm a fan of ttl as it's
>> application specific (producers get to chose how long their messages
>> should live) and provides a reasonable heuristic for deleting messages
>> that aren't consumed in a timely manner.
>
> And as it happens, ttl is one of the key controls of reliable delivery: in
> order to achieve reliable delivery, you need to be able to bound the lifetime
> of a packet in the network, so that duplicates don't outlive their
> deduplication buffer entries.
>
> I think implementing ttl is important. We could build a dumb form of it quickly
> enough, where as a message is considered for delivery its ttl is checked, but
> the real challenge lies in collecting expired messages while they're idle.
> Per-message ttls aggravate the problem: if only it were a per *queue* ttl, it'd
> be so much easier!
>
> Tony
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



--
Dan Di Spaltro

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

per-queue message TTLs [was Re: Bound Queues]

Matthias Radestock-3
Dan Di Spaltro wrote:
> On Wed, Apr 14, 2010 at 4:13 AM, Tony Garnock-Jones <[hidden email]> wrote:
>> if only it were a per *queue* ttl, it'd be so much easier!
> ttl's on a queue level would be very useful to us

That's a really interesting idea, and it is a lot easier to implement
than per-message TTLs.

So here is a question to all RabbitMQ users who have been asking /
secretly wishing for TTLs: Would per-queue TTLs, i.e. all messages
getting the same TTL when they are enqueued, be good enough (or even
preferable) for you? Or do you definitely need to be able to specify
TTLs on per-message basis (and if so, what is the use case)?


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: per-queue message TTLs [was Re: Bound Queues]

Bryan Murphy-4
On Wed, Apr 14, 2010 at 1:13 PM, Matthias Radestock <[hidden email]> wrote:
Dan Di Spaltro wrote:
> On Wed, Apr 14, 2010 at 4:13 AM, Tony Garnock-Jones <[hidden email]> wrote:
>> if only it were a per *queue* ttl, it'd be so much easier!
> ttl's on a queue level would be very useful to us

That's a really interesting idea, and it is a lot easier to implement
than per-message TTLs.

So here is a question to all RabbitMQ users who have been asking /
secretly wishing for TTLs: Would per-queue TTLs, i.e. all messages
getting the same TTL when they are enqueued, be good enough (or even
preferable) for you? Or do you definitely need to be able to specify
TTLs on per-message basis (and if so, what is the use case)?


We'd find per-queue TTLs very useful.

Bryan

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: per-queue message TTLs [was Re: Bound Queues]

Garrett Smith-5
In reply to this post by Matthias Radestock-3
On Wed, Apr 14, 2010 at 1:13 PM, Matthias Radestock
<[hidden email]> wrote:

> Dan Di Spaltro wrote:
>> On Wed, Apr 14, 2010 at 4:13 AM, Tony Garnock-Jones <[hidden email]> wrote:
>>> if only it were a per *queue* ttl, it'd be so much easier!
>> ttl's on a queue level would be very useful to us
>
> That's a really interesting idea, and it is a lot easier to implement
> than per-message TTLs.
>
> So here is a question to all RabbitMQ users who have been asking /
> secretly wishing for TTLs: Would per-queue TTLs, i.e. all messages
> getting the same TTL when they are enqueued, be good enough (or even
> preferable) for you? Or do you definitely need to be able to specify
> TTLs on per-message basis (and if so, what is the use case)?

Per message ttl and per queue ttl strike me as being very different
and both having valid use cases.

Per message ttl is specified based on the producer's requirement or
vantage point. E.g. a caller might designate a request as being valid
only for a certain amount of time (e.g. request price quote) -- or the
message content is no longer valid after the expiration (e.g. send
price quote). In such cases queues have nothing to say about ttl.

A per queue ttl would apply to consumers. E.g. a consumer might not
fulfill a request because it's become "stale".

For my part, per queue ttls would be preferable. I want my consumers
to be able to drop off, come back and handle requests that aren't
"stale" (basically to handle restarts, though there are other
scenarios). In my case, the consumers know what stale means better
than the producers, though that's obviously application specific.

I see both being reasonable features that are independent of one another.

Garrett

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: per-queue message TTLs [was Re: Bound Queues]

alex chen-13
In reply to this post by Matthias Radestock-3
> TTLs: Would per-queue TTLs, i.e. all messages
> getting the same TTL when they
> are enqueued, be good enough (or even
> preferable) for you?

We prefer to have per queue TTL.  It would also be very useful if rabbit can support limiting max number of messages per queue, and/or max num of bytes per queue.  When the queue length reached the limit, drop old messages.

-alex


     

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: per-queue message TTLs [was Re: Bound Queues]

Al Tobey
In reply to this post by Matthias Radestock-3
On Wed, Apr 14, 2010 at 11:13 AM, Matthias Radestock <[hidden email]> wrote:
Dan Di Spaltro wrote:
> On Wed, Apr 14, 2010 at 4:13 AM, Tony Garnock-Jones <[hidden email]> wrote:
>> if only it were a per *queue* ttl, it'd be so much easier!
> ttl's on a queue level would be very useful to us

That's a really interesting idea, and it is a lot easier to implement
than per-message TTLs.

So here is a question to all RabbitMQ users who have been asking /
secretly wishing for TTLs: Would per-queue TTLs, i.e. all messages
getting the same TTL when they are enqueued, be good enough (or even
preferable) for you? Or do you definitely need to be able to specify
TTLs on per-message basis (and if so, what is the use case)?

Per-queue would be far more useful IMO.   Display the TTL's in rabbitmqctl list_queues and in the rabbitmq-status plugin and it'll be really easy to allow users to figure out why their messages are disappearing after $TTL seconds.   It also makes the life of the administrator easier, since it would be visible without peeking at messages.

-Al

Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
srw
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

srw
In reply to this post by Tony Garnock-Jones-2
Tony Garnock-Jones-2 wrote
Dan Di Spaltro wrote:
> Is there anyway you can specify a memory/disk/length limit to a queue
> size with the action being a similar to a fixed window?

Not at the moment, no.

> If not, is there any chance it is on the roadmap?  I know its not in
> the amqp spec, but it could be treated like a header.

It's not exactly on the roadmap
...
...
...
As a RabbitMQ newbie I was thinking on this issue too: is it possible to block publisher's send while the queue is above a certain threshold?

I am experiencing many RabbitMQ crashes because the queue size surpass the machine memory. This is happening in many scenarios where the ratio between #published_messages/#consumed_messages is very high (and the message size is big). Don't know if it's very uncommon. My use case is a long text mining pipeline where there is a lot of input data and some processes cause a bottleneck (but with the solution exposed above the bottleneck will just slow the publishers and every process will finish).

Thanks,
srw
srw
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

srw
In reply to this post by Tony Garnock-Jones-2
Tony Garnock-Jones-2 wrote
Dan Di Spaltro wrote:
> Is there anyway you can specify a memory/disk/length limit to a queue
> size with the action being a similar to a fixed window?

Not at the moment, no.

> If not, is there any chance it is on the roadmap?  I know its not in
> the amqp spec, but it could be treated like a header.

It's not exactly on the roadmap
...
...
...
As a RabbitMQ newbie I was thinking on this issue too: is it possible to block publisher's send while the queue is above a certain threshold?

I am experiencing many RabbitMQ crashes because the queue size surpass the machine memory. This is happening in many scenarios where the ratio between #published_messages/#consumed_messages is very high (and the message size is big). Don't know if it's very uncommon. My use case is a long text mining pipeline where there is a lot of input data and some processes cause a bottleneck (but with the solution exposed above the bottleneck will just slow the publishers and every process will finish).

Thanks,
srw
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Colin Clark-5
Are you not able to add more consumers?

On 5/12/2010 11:47 AM, srw wrote:

Tony Garnock-Jones-2 wrote:
  
Dan Di Spaltro wrote:
    
Is there anyway you can specify a memory/disk/length limit to a queue
size with the action being a similar to a fixed window?
      
Not at the moment, no.

    
If not, is there any chance it is on the roadmap?  I know its not in
the amqp spec, but it could be treated like a header.
      
It's not exactly on the roadmap
...
...
...

    
As a RabbitMQ newbie I was thinking on this issue too: is it possible to
block publisher's send while the queue is above a certain threshold?

I am experiencing many RabbitMQ crashes because the queue size surpass the
machine memory. This is happening in many scenarios where the ratio between
#published_messages/#consumed_messages is very high (and the message size is
big). Don't know if it's very uncommon. My use case is a long text mining
pipeline where there is a lot of input data and some processes cause a
bottleneck (but with the solution exposed above the bottleneck will just
slow the publishers and every process will finish).

Thanks,
srw

  

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
srw
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

srw
Colin Clark-5 wrote
Are you not able to add more consumers?
Theoretically yes but not in this case because the publisher speed is very high and the overall throughput is fine, so there isn't a real need for more subscribers. I can put an sleep there... or do another trick but don't feel comfortably with that solution.

I think Weblogic JMS (and multithreading Python Queues too) provides this kind of feature. Don't know if formally this is right or wrong.

Thanks,
srw
Reply | Threaded
Open this post in threaded view
|

Re: Bound Queues

Tony Garnock-Jones-2
In reply to this post by srw
srw wrote:
> As a RabbitMQ newbie I was thinking on this issue too: is it possible to
> block publisher's send while the queue is above a certain threshold?

Hmm. Well there's a builtin feature that if memory usage gets above a threshold
for the server *as a whole*, a "channel.flow" notification (like XOFF) gets
sent out to connected clients until the backlog clears. It's enabled by
default, though, so maybe something's wrong. Which client library are you
using? Some of them don't pay attention to channel.flow, in violation of the
spec, which can lead to overflowing the server.

Tony


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
12