Publishing Performance

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Publishing Performance

Umutcan
Hi,

Does publishing get slower while number of the queued messages are
increasing?

While I am doing an operation, publishing speed get slower in time.
Operation publishes a certain amount of messages to RMQ and then  a
consumer processes messages. Since publishing speed is higher than
consuming speed, messages waits for a while in the queue. I am not
running operation again until queue is empty.

Thanks,
Umutcan
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Publishing Performance

Michael Klishin-2
On 4 Feb 2014, at 11:24, Umutcan <[hidden email]> wrote:

> Does publishing get slower while number of the queued messages are increasing?
>
> While I am doing an operation, publishing speed get slower in time. Operation publishes a certain amount of messages to RMQ and then  a consumer processes messages. Since publishing speed is higher than consuming speed, messages waits for a while in the queue. I am not running operation again until queue is empty.

If consumers do not keep up with publishers, eventually RabbitMQ will throttle publishers
until consumers drain enough messages to eliminate RAM and disk space pressure.

See http://rabbitmq.com/memory.html.

OS swapping can also take place, which affects performance severely.

Otherwise there is little connection between publisher throughput and how many
messages there are in the queues.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Publishing Performance

Simon MacMullen-2
On 04/02/2014 7:30AM, Michael Klishin wrote:
> On 4 Feb 2014, at 11:24, Umutcan <[hidden email]> wrote:
>
>> Does publishing get slower while number of the queued messages are increasing?
>>
>> While I am doing an operation, publishing speed get slower in time. Operation publishes a certain amount of messages to RMQ and then  a consumer processes messages. Since publishing speed is higher than consuming speed, messages waits for a while in the queue. I am not running operation again until queue is empty.
>
> If consumers do not keep up with publishers, eventually RabbitMQ will throttle publishers
> until consumers drain enough messages to eliminate RAM and disk space pressure.

First of all, RabbitMQ will start paging messages out to disk. Since at
all times RabbitMQ tries not to accept messages faster than it can do
something sensible with them, this will mean the publishing rate goes
down substantially at this point.

You can see if a queue is paging to disk in the management plugin web
UI, or by looking at backing_queue_status.ram_msg_count in rabbitmqctl
list_queues or the HTTP API.

Eventually if the disk fills up, or something else happens to make
memory usage spike, then the memory alarm can go off and block
publishers completely until memory use drops again.

> See http://rabbitmq.com/memory.html.
>
> OS swapping can also take place, which affects performance severely.

We try to avoid that, but on a machine where other services are running
it's not always possible.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Publishing Performance

Simone Sciarrati
Hi,

I tried rabbitmqctl list_queues and could not see
backing_queue_status.ram_msg_count, I think you mean ./rabbitmqadmin
list queues name backing_queue_status.ram_msg_count (or variations of
it)

How does backing_queue_status.ram_msg_count relate to the queue paging
messages to disk? I have seen a few threads and in one it was
suggested to look at the comments in
https://github.com/michaelklishin/rabbit-hole/blob/master/queues.go#L10
but there I see:

RAMMessageCount       int64   `json:"ram_msg_count"`
// Number of outstanding acks held in RAM

It is not clear to me how 'number of outstanding acks held in RAM'
relates to messages being paged to disk. Also where does it show in
the UI?

Thanks,

S

On Tue, Feb 4, 2014 at 11:24 AM, Simon MacMullen <[hidden email]> wrote:

> On 04/02/2014 7:30AM, Michael Klishin wrote:
>>
>> On 4 Feb 2014, at 11:24, Umutcan <[hidden email]> wrote:
>>
>>> Does publishing get slower while number of the queued messages are
>>> increasing?
>>>
>>> While I am doing an operation, publishing speed get slower in time.
>>> Operation publishes a certain amount of messages to RMQ and then  a consumer
>>> processes messages. Since publishing speed is higher than consuming speed,
>>> messages waits for a while in the queue. I am not running operation again
>>> until queue is empty.
>>
>>
>> If consumers do not keep up with publishers, eventually RabbitMQ will
>> throttle publishers
>> until consumers drain enough messages to eliminate RAM and disk space
>> pressure.
>
>
> First of all, RabbitMQ will start paging messages out to disk. Since at all
> times RabbitMQ tries not to accept messages faster than it can do something
> sensible with them, this will mean the publishing rate goes down
> substantially at this point.
>
> You can see if a queue is paging to disk in the management plugin web UI, or
> by looking at backing_queue_status.ram_msg_count in rabbitmqctl list_queues
> or the HTTP API.
>
> Eventually if the disk fills up, or something else happens to make memory
> usage spike, then the memory alarm can go off and block publishers
> completely until memory use drops again.
>
>
>> See http://rabbitmq.com/memory.html.
>>
>> OS swapping can also take place, which affects performance severely.
>
>
> We try to avoid that, but on a machine where other services are running it's
> not always possible.
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Publishing Performance

Simon MacMullen-2
On 05/02/2014 12:10AM, Simone Sciarrati wrote:
> I tried rabbitmqctl list_queues and could not see
> backing_queue_status.ram_msg_count, I think you mean ./rabbitmqadmin
> list queues name backing_queue_status.ram_msg_count (or variations of
> it)

You can do that too. For rabbitmqctl you need "rabbitmqctl list_queues
name backing_queue_status" and then pull ram_msg_count out yourself.

> How does backing_queue_status.ram_msg_count relate to the queue paging
> messages to disk?

It tells you the number of messages the queue has in RAM. If that number
is the same as the total message count, no paging has taken place.
Otherwise it has.

> I have seen a few threads and in one it was
> suggested to look at the comments in
> https://github.com/michaelklishin/rabbit-hole/blob/master/queues.go#L10
> but there I see:
>
> RAMMessageCount       int64   `json:"ram_msg_count"`
> // Number of outstanding acks held in RAM

You need the comment before that line, not after it:

        // Number of messages held in RAM
        RAMMessageCount       int64   `json:"ram_msg_count"`

> Also where does it show in the UI?

On the right hand side of "Details" under "Overview" on the queue
details page.

It was only added to the UI in 3.2.0 BTW.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss