Active consumers in declare-ok is less than the actual number of consumers

[Gerolf Seitz]

Hi,

When passively declaring a queue, the returned consumer count is not the same as the consumer count via the management plugin / http API. Turns out, rabbitmq returns the number of consumers that are ready to receive a message.

As an example: there are 5 (manually acknowledging) consumers, 3 of which have consumed a message from the queue, but have not sent an ACK for the respective messages to the broker. Declaring a queue now returns a declare-ok with consumer-count == 2, whereas I would have expected it to be 5.

Looking at the rabbitmq-server code reveals that the "active consumer" part of the state is used for getting the next consumer for round robin delivery and returning the number of consumers as part of the declare-ok message.

Since I don't suspect this to be an oversight but rather a conscious decision, I'm interested in the rationale behind it. I'm aware that changing the semantics at this point is probably not possible and I may have to resort to fetching the consumer count via the http api (which brings some organizational overhead).

Can anybody shed some light on this?

Cheers,

  Gerolf

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Simon MacMullen-2]

On 20/03/12 06:43, Gerolf Seitz wrote:

> When passively declaring a queue, the returned consumer count is not the
> same as the consumer count via the management plugin / http API. Turns
> out, rabbitmq returns the number of consumers that are ready to receive
> a message.
>
> As an example: there are 5 (manually acknowledging) consumers, 3 of
> which have consumed a message from the queue, but have not sent an ACK
> for the respective messages to the broker. Declaring a queue now returns
> a declare-ok with consumer-count == 2, whereas I would have expected it
> to be 5.
>
> Looking at the rabbitmq-server code reveals that the "active consumer"
> part of the state is used for getting the next consumer for round robin
> delivery and returning the number of consumers as part of the declare-ok
> message.
>
> Since I don't suspect this to be an oversight but rather a conscious
> decision, I'm interested in the rationale behind it. I'm aware that
> changing the semantics at this point is probably not possible and I may
> have to resort to fetching the consumer count via the http api (which
> brings some organizational overhead).
>
> Can anybody shed some light on this?

Hi Gerolf.

Yes, this is deliberate. Code archaeology yielded this:

http://hg.rabbitmq.com/rabbitmq-server/rev/32e357ad03d6

"count queue consumers as required by the spec".

The spec actually says:

     Reports the number of active consumers for the queue. Note that
     consumers can  suspend activity (Channel.Flow) in which case they
     do not appear in this count.

So that is slightly ambiguous - we are excluding consumers that have
issued channel.flow, but also those which are blocked due to qos / ack
interaction. But those could easily be defined as "inactive"...

Hmm. I'll have a think about this. What are you trying to do?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Gerolf Seitz]

On Tue, Mar 20, 2012 at 12:17 PM, Simon MacMullen <[hidden email]> wrote:

On 20/03/12 06:43, Gerolf Seitz wrote:

When passively declaring a queue, the returned consumer count is not the
same as the consumer count via the management plugin / http API. Turns
out, rabbitmq returns the number of consumers that are ready to receive
a message.

As an example: there are 5 (manually acknowledging) consumers, 3 of
which have consumed a message from the queue, but have not sent an ACK
for the respective messages to the broker. Declaring a queue now returns
a declare-ok with consumer-count == 2, whereas I would have expected it
to be 5.

Looking at the rabbitmq-server code reveals that the "active consumer"
part of the state is used for getting the next consumer for round robin
delivery and returning the number of consumers as part of the declare-ok
message.

Since I don't suspect this to be an oversight but rather a conscious
decision, I'm interested in the rationale behind it. I'm aware that
changing the semantics at this point is probably not possible and I may
have to resort to fetching the consumer count via the http api (which
brings some organizational overhead).

Can anybody shed some light on this?

Hi Gerolf.

Yes, this is deliberate. Code archaeology yielded this:

http://hg.rabbitmq.com/rabbitmq-server/rev/32e357ad03d6

"count queue consumers as required by the spec".

The spec actually says:

   Reports the number of active consumers for the queue. Note that
   consumers can  suspend activity (Channel.Flow) in which case they
   do not appear in this count.

So that is slightly ambiguous - we are excluding consumers that have issued channel.flow, but also those which are blocked due to qos / ack interaction. But those could easily be defined as "inactive"...

Hmm. I'll have a think about this. What are you trying to do?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

The reply-to-author setting of rabbitmq-discuss screwed me ;)

Hi,

The use case is this:

We have a pool of consumers (each running in a heavy weight process) that can only process a single unit of work at a time each.

The messages that are sent to these consumers contain data that can become "old" quite quickly (think stock ticks), so queuing hundreds of messages is not an option.

Also, these consumers need to be kept busy all the time: as soon as a unit of work is done, the next task will be submitted and picked up by the free consumer. (prefetch=1, autoack=false).

Based on the number of consumers, the system decides which unit of work is sent next, mostly based on the previous time the unit of work took to complete.

Example:

7 consumers overall

Per configuration, only up to 4 consumers may be used to process really long running tasks (eg. >1 minute task duration)

The rest of the consumers is used for short running tasks. (eg. <1 minute)

The reason why we don't have 2 different queues is twofold:

1) if there are no long running tasks to be executed, all consumers should be used for short running tasks

2) the decision whether a task is short or long running is made by the system that sends the messages to the broker and the decision process can be tweaked via configuration. This is easier to manipulate than reconfiguring and restarting the heavy weight consumer processes.

I hope this clears it up a bit ;)

Cheers

--

Gerolf Seitz

twitter: @gersei

code: github.com/gseitz

blog: www.gerolfseitz.com

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss