queue_procs question

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

queue_procs question

Sean Allen
I'm trying to understand what I'm seeing with memory usage in our cluster.

We have a mirrored queue that we sometimes have large amounts of publishes come in during a relatively short period of time. (Say 2 million messages during 5-10 minutes)

While we are publishing, memory usage on both the primary and mirrored nodes rises greatly. Almost all the memory used is queue_procs and binary. Once publishing ceases, the primary node returns to an amount of memory usage in line with the message size (whereas during publishing its 5 times or so what you would expect just based on message size).

The mirrored queue trails behind in terms of memory usage dropping but eventually levels out to similar usage some time later (couple hours later).

If I turn off the consumer of the queue, the memory still stabilizes eventually.

This has led us to believe that the queue_procs and binary memory usage is related to publishing and to moving data to the mirror on the other node. Is that a reasonable assumption?

--

Ce n'est pas une signature

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Simon MacMullen-2
I'm not quite sure what the real question is here. The queue_procs and
binary memory slices will describe memory used by queues (with message
bodies in 'binary' and everything else in 'queue_procs'). This is a
fairly raw count according to what the Erlang VM tells us, so for
example this includes memory used by unreachable objects that have not
yet been GCed.

So "related to publishing and to moving data to the mirror on the other
node" - well, yes, those are things queues do, so they can use memory
doing so. But they can do other things too, and thus might use memory
for other purposes :-)

Cheers, Simon

On 13/05/2014 17:31, Sean Allen wrote:

> I'm trying to understand what I'm seeing with memory usage in our cluster.
>
> We have a mirrored queue that we sometimes have large amounts of
> publishes come in during a relatively short period of time. (Say 2
> million messages during 5-10 minutes)
>
> While we are publishing, memory usage on both the primary and mirrored
> nodes rises greatly. Almost all the memory used is queue_procs and
> binary. Once publishing ceases, the primary node returns to an amount of
> memory usage in line with the message size (whereas during publishing
> its 5 times or so what you would expect just based on message size).
>
> The mirrored queue trails behind in terms of memory usage dropping but
> eventually levels out to similar usage some time later (couple hours later).
>
> If I turn off the consumer of the queue, the memory still stabilizes
> eventually.
>
> This has led us to believe that the queue_procs and binary memory usage
> is related to publishing and to moving data to the mirror on the other
> node. Is that a reasonable assumption?

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Sean Allen
Let me ask in a different way:

Why would a node use more memory in binary and queue_procs while publishing to an exchange and then have that memory drop dramatically without taking any messages off of the queue? And by dramatically I mean 5-6x less memory within a few minutes of publishing ending.




On Wed, May 14, 2014 at 4:05 AM, Simon MacMullen <[hidden email]> wrote:
I'm not quite sure what the real question is here. The queue_procs and binary memory slices will describe memory used by queues (with message bodies in 'binary' and everything else in 'queue_procs'). This is a fairly raw count according to what the Erlang VM tells us, so for example this includes memory used by unreachable objects that have not yet been GCed.

So "related to publishing and to moving data to the mirror on the other node" - well, yes, those are things queues do, so they can use memory doing so. But they can do other things too, and thus might use memory for other purposes :-)

Cheers, Simon


On 13/05/2014 17:31, Sean Allen wrote:
I'm trying to understand what I'm seeing with memory usage in our cluster.

We have a mirrored queue that we sometimes have large amounts of
publishes come in during a relatively short period of time. (Say 2
million messages during 5-10 minutes)

While we are publishing, memory usage on both the primary and mirrored
nodes rises greatly. Almost all the memory used is queue_procs and
binary. Once publishing ceases, the primary node returns to an amount of
memory usage in line with the message size (whereas during publishing
its 5 times or so what you would expect just based on message size).

The mirrored queue trails behind in terms of memory usage dropping but
eventually levels out to similar usage some time later (couple hours later).

If I turn off the consumer of the queue, the memory still stabilizes
eventually.

This has led us to believe that the queue_procs and binary memory usage
is related to publishing and to moving data to the mirror on the other
node. Is that a reasonable assumption?




--

Ce n'est pas une signature

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Simon MacMullen-2
On 14/05/14 15:42, Sean Allen wrote:
> Let me ask in a different way:
>
> Why would a node use more memory in binary and queue_procs while
> publishing to an exchange and then have that memory drop dramatically
> without taking any messages off of the queue? And by dramatically I mean
> 5-6x less memory within a few minutes of publishing ending.

The messages could be getting paged out? See
http://www.rabbitmq.com/memory.html#memsup-paging.

Alternatively, it could just be that a lot of garbage is getting created
while publishing, and it take that long to collect it.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Sean Allen
According to the UI, it isnt messages getting paged out.

If it is the latter, would it make sense to see the same issue on mirrored versions of the queue but see if take much longer for that node to return to a "normal" level of memory usage?




On Thu, May 15, 2014 at 6:36 AM, Simon MacMullen <[hidden email]> wrote:
On 14/05/14 15:42, Sean Allen wrote:
Let me ask in a different way:

Why would a node use more memory in binary and queue_procs while
publishing to an exchange and then have that memory drop dramatically
without taking any messages off of the queue? And by dramatically I mean
5-6x less memory within a few minutes of publishing ending.

The messages could be getting paged out? See http://www.rabbitmq.com/memory.html#memsup-paging.

Alternatively, it could just be that a lot of garbage is getting created while publishing, and it take that long to collect it.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal



--

Ce n'est pas une signature

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Matthias Radestock-3
On 15/05/14 16:05, Sean Allen wrote:
> would it make sense to see the same issue on mirrored versions of the
> queue but see if take much longer for that node to return to a
> "normal" level of memory usage?

Yes, since queue slaves can "churn" more binary data than masters.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue_procs question

Sean Allen



On Tue, May 20, 2014 at 5:44 AM, Matthias Radestock <[hidden email]> wrote:
On 15/05/14 16:05, Sean Allen wrote:
would it make sense to see the same issue on mirrored versions of the
queue but see if take much longer for that node to return to a
"normal" level of memory usage?

Yes, since queue slaves can "churn" more binary data than masters.

Matthias.


Thanks Mathias.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss