"Other process memory" growth

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

"Other process memory" growth

carlhoerberg
Have a 3 node cluster, normally it hovers around 200-300mb of ram, but suddenly the "Other process memory" started to grow:

https://www.dropbox.com/s/lhnjvnzf2rilwox/Screen%20Shot%202013-03-23%20at%2010.32.43.png

I restarted it (when it reached 5GB) but then the other nodes also raised their "Other process memory" use too. What may be the reason for that? The docs (http://www.rabbitmq.com/memory-use.html) aren't too clear to me.

I can see a very high deliver/redeliver rate, compared to publish rate, may that be a clue?

RabbitMQ 3.0.4, Erlang R14B04.
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

Emile Joubert

Hi Carl,

On 23/03/13 02:47, carlhoerberg wrote:
> Have a 3 node cluster, normally it hovers around 200-300mb of ram, but
> suddenly the "Other process memory" started to grow:


"Other" is all use left after connections, queues and plugins are
accounted for. This could be due to internal Erlang VM processes
consuming memory. One way to dig further is to look at the Erlang
processes using the most memory:

$ etop -sort memory

More information about use of etop is available here:
http://www.erlang.org/doc/man/etop.html




-Emile









_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

Matthias Radestock-3
On 26/03/13 12:26, Emile Joubert wrote:
> One way to dig further is to look at the Erlang
> processes using the most memory:
>
> $ etop -sort memory

Or

rabbitmqctl eval '[[{pid, R}, M | process_info(R)] || {M, R} <-
lists:sublist(lists:reverse(lists:sort([{process_info(P, memory), P} ||
P <- erlang:processes()])), 10)].'


Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

carlhoerberg
Thank you!
Here's the output: https://www.dropbox.com/s/spob20fzupi14tz/rmqmem.txt
Must admit that it doesn't make much sense to me :P Do you have any clue what's taking all that memory?



On Tuesday 26 March 2013 at 20:51, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 26/03/13 12:26, Emile Joubert wrote:
> > One way to dig further is to look at the Erlang
> > processes using the most memory:
> >
> > $ etop -sort memory
>
> Or
>
> rabbitmqctl eval '[[{pid, R}, M | process_info(R)] || {M, R} <-
> lists:sublist(lists:reverse(lists:sort([{process_info(P, memory), P} ||
> P <- erlang:processes()])), 10)].'
>
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25725&i=0)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/Other-process-memory-growth-tp25675p25725.html 
> To unsubscribe from "Other process memory" growth, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)



Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

Matthias Radestock-3
On 28/03/13 07:32, carlhoerberg wrote:
> Here's the output: https://www.dropbox.com/s/spob20fzupi14tz/rmqmem.txt
> Must admit that it doesn't make much sense to me :P Do you have any clue
> what's taking all that memory?

What does the Overview page in the management UI show? Should be pretty
obvious from that.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

carlhoerberg

No..? What do you mean?

On Mar 28, 2013 4:09 PM, "Matthias Radestock-3 [via RabbitMQ]" <[hidden email]> wrote:
On 28/03/13 07:32, carlhoerberg wrote:
> Here's the output: https://www.dropbox.com/s/spob20fzupi14tz/rmqmem.txt
> Must admit that it doesn't make much sense to me :P Do you have any clue
> what's taking all that memory?

What does the Overview page in the management UI show? Should be pretty
obvious from that.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



If you reply to this email, your message will be added to the discussion below:
http://rabbitmq.1065348.n5.nabble.com/Other-process-memory-growth-tp25675p25756.html
To unsubscribe from "Other process memory" growth, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

Matthias Radestock-3
On 28/03/13 08:11, carlhoerberg wrote:
> No..? What do you mean?

Can you post it? I do have a theory, but it could be wrong; that's why I
wanted to check what the overview page shows.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

carlhoerberg

Ok :) https://www.dropbox.com/sc/mb7zw9gqsrxg7to/H0x1QYwfau

On Mar 28, 2013 4:14 PM, "Matthias Radestock-3 [via RabbitMQ]" <[hidden email]> wrote:
On 28/03/13 08:11, carlhoerberg wrote:
> No..? What do you mean?

Can you post it? I do have a theory, but it could be wrong; that's why I
wanted to check what the overview page shows.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



If you reply to this email, your message will be added to the discussion below:
http://rabbitmq.1065348.n5.nabble.com/Other-process-memory-growth-tp25675p25758.html
To unsubscribe from "Other process memory" growth, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: "Other process memory" growth

Emile Joubert

Hi Carl,

On 28/03/13 09:06, carlhoerberg wrote:
> Ok :) https://www.dropbox.com/sc/mb7zw9gqsrxg7to/H0x1QYwfau

Would you be able to provide some more details for us to investigate
this further? We will need to logfiles and the sasl logfiles from all 3
nodes. Please provide entries from at least a while before the memory
use on the one node started to increase.

Then we'd like you to execute a command on the node with abnormal memory
(tiger01), but take some precautions first. If you are able to bring
more swapspace online then please do so. If there are any other
non-critical processes on the machine then please stop them, as this
command has the potential to impose heavy load. Please be aware that
this command could cause the node to crash. If you have HA redundancy in
place then it shouldn't matter, but if you are not comfortable with this
prospect then don't proceed.

Otherwise please run this command:

rabbitmqctl eval 'io:put_chars(standard_error, "this should appear in
startup_err\n").'

and check that the expected text appeared in startup_err (normally in
/var/log/rabbitmq). If the text did not appear then do not proceed.

Otherwise please run this command:

rabbitmqctl eval 'erlang:process_display(element(2,
hd(lists:reverse(lists:sort([{process_info(P, memory), P} ||  P <-
erlang:processes()])))), backtrace).'

The command will return and the startup_err file should start growing
soon afterwards. It could take a long time until it stops growing. When
it does stop growing please send it to [hidden email] along with
the logfiles, preferably compressed.




-Emile







_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss