Rabbitmq 3.2.4 running out of memory
Locked
12
12
Rabbitmq 3.2.4 running out of memory
 
|
Hi
Can someone please let us know wht we need to do ? we are getting out of memory errors in the rabbitmq 3.2.4 when we are trying to access rabbitmq on port 15672 using connections api . The script calls curl via shell console.
curl -i -s --connect-timeout 5 -u admin:changeme http://$hostname1:15672/api/connections?columns=node,name This curl call is made from many hosts at same to get the number of connections running on rabbitmq and we see the connections are blocked on this port.
netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c 3 CLOSE_WAIT 1882 ESTABLISHED 1 LISTEN 10 TIME_WAIT
Thanks Srikanth Logs show this: vm_memory_high_watermark set. Memory used:14888001288 allowed:11753753804
=WARNING REPORT==== 26-Aug-2014::05:58:18 === memory resource limit alarm set on node rabbit@adsmv0019. ********************************************************** *** Publishers will be blocked until this alarm clears ***
********************************************************** =INFO REPORT==== 26-Aug-2014::05:58:20 === vm_memory_high_watermark clear. Memory used:8686452200 allowed:11753753804
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:
> > Logs show this: > > vm_memory_high_watermark set. Memory used:14888001288 > allowed:11753753804 > > =WARNING REPORT==== 26-Aug-2014::05:58:18 > === > memory resource limit alarm set on node rabbit@adsmv0019. > > ********************************************************** > *** Publishers will be blocked until this alarm clears *** > ********************************************************** > > > =INFO REPORT==== 26-Aug-2014::05:58:20 > === > vm_memory_high_watermark clear. Memory used:8686452200 > allowed:11753753804 Use rabbitmqctl report or management UI (node info page) to find out what takes memory. Then either eliminate that thing (likely it is messages in the queue, there is a fixed RAM cost per message: the queue index entry(-ies)) or bump the watermark. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
In reply to this post by srikanth tns
On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:
> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c > 3 CLOSE_WAIT > 1882 ESTABLISHED > 1 LISTEN > 10 TIME_WAIT That's about 1900 concurrent connections to a single node. Without knowing how many channels can be open on the open ones, that's still a fairly large number for a single node. Consider clustering this node with a couple more and spread the connections between them, e.g. using an HAproxy instance in front of the cluster. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
In reply to this post by srikanth tns
On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:
> > Thanks > Srikanth Also: this list has moved to rabbitmq-users: https://groups.google.com/forum/#!forum/rabbitmq-users Please start all new discussions there. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
In reply to this post by Michael Klishin-4
The UI is not accessible , The 3-nodes are clustered with HA two policy rabbitmqctl reports shows Reporting server status on {{2014,8,26},{18,39,31}} ... Status of node rabbit@adsmv0020 ... [{pid,22153}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},
{rabbitmq_management,"RabbitMQ Management Console","3.2.4"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"},
{os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"}, {sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,2285543720}, {connection_procs,207275088}, {queue_procs,599282104}, {plugins,234739776}, {other_proc,309911344},
{mnesia,289075584}, {mgmt_db,9096}, {msg_index,14688048}, {other_ets,28236624}, {binary,290512400}, {code,17776116}, {atom,1621321},
{other_system,292416219}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19386892288}, {file_descriptors,
[{total_limit,327579}, {total_used,3137}, {sockets_limit,294819}, {sockets_used,3135}]}, {processes,[{limit,1048576},{used,144321}]}, {run_queue,0},
{uptime,6267}] Cluster status of node rabbit@adsmv0020 ... [{nodes,[{disc,[rabbit@adsmv0019,rabbit@adsmv0018]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0018,rabbit@adsmv0019,rabbit@adsmv0020]},
{partitions,[]}] Application environment of node rabbit@adsmv0020 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,
rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,
sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array,
rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0020-plugins-expand"},
{queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true}, {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}}, {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},
{server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary,
{packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}},
{exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}]
Status of node rabbit@adsmv0019 ... [{pid,27139}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"}, {os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},
{xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"},
{sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,2375638736},
{connection_procs,228104952}, {queue_procs,643238504}, {plugins,259036872}, {other_proc,276878920}, {mnesia,288547136}, {mgmt_db,9096},
{msg_index,14503600}, {other_ets,30000664}, {binary,321298960}, {binary,321298960}, {code,17776116}, {atom,1621321},
{other_system,294622595}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19439706112}, {file_descriptors,
[{total_limit,327579}, {total_used,3466}, {sockets_limit,294819}, {sockets_used,3464}]}, {processes,[{limit,1048576},{used,159459}]}, {run_queue,0},
{uptime,6302}] Cluster status of node rabbit@adsmv0019 ... [{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0018,rabbit@adsmv0019]},
{partitions,[]}] Application environment of node rabbit@adsmv0019 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue,
rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue,
rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,
file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]},
{log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},
{plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0019-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true}, {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},
{server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary,
{packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}},
{exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}]
Status of node rabbit@adsmv0018 ... [{pid,20727}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"}, {os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},
{xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"},
{sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,6095435856},
{connection_procs,185360136}, {queue_procs,526782896}, {plugins,-1872186488}, {other_proc,2447546120}, {mnesia,289695904}, {mgmt_db,3870226464},
{msg_index,23253920}, {other_ets,51366128}, {binary,249989160}, {code,17776116}, {atom,1622129}, {other_system,304003371}]},
{vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19338166272}, {file_descriptors, [{total_limit,327579},
{total_used,2748}, {sockets_limit,294819}, {sockets_used,2746}]}, {processes,[{limit,1048576},{used,126430}]}, {run_queue,0}, {uptime,6789}]
Cluster status of node rabbit@adsmv0018 ... [{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0019,rabbit@adsmv0018]},
{partitions,[]}] Application environment of node rabbit@adsmv0018 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,
rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,
sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array,
rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0018-plugins-expand"},
{queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true}, {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}}, {server_properties,[]},
{ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw},
{reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]},
{tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}] On Tue, Aug 26, 2014 at 11:42 AM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
|
|
Looking at your memory: {memory, [{total,<a href="tel:6095435856" value="+16095435856" target="_blank">6095435856},
{connection_procs,185360136}, 185MB {queue_procs,526782896}, 526MB
{plugins,-1872186488}, {other_proc,2447546120}, 2447MB {mnesia,289695904}, 290MB
{mgmt_db,3870226464},3870MB {msg_index,23253920},23MB
{other_ets,51366128},51MB {binary,249989160},250MB {code, 17776116},17MB
{atom,1622129}, 1MB {other_system,304003371}304MB It appears your available memory is about 12GB available, .07 * of that would mean about 8GB available, and you're using right at that from all I'm seeing. I could be mis-understanding a parameter. But it looks like you have a LOT of queues, and then a LOT of statistics on those queues - perhaps reducing the statistics gathering rate or some other tuning might help?
Jason On Tue, Aug 26, 2014 at 1:47 PM, srikanth tns <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 26 August 2014 at 22:59:09, Jason McIntosh ([hidden email]) wrote:
> {mgmt_db,3870226464},3870MB Thanks, Jason. So here's our key offender. Besides memory leaks that are fixed in the statistics DB from time to time, you can turn off fine grained stats. This should drastically reduce the amount of RAM used by the stats database. See http://rabbitmq.com/management.html. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
what are the changes we see on the UI statistics ? I have applied it on our test instances and see one of the main charts for message rates is gone. Currently we are servicing 10800 nodes on rabbitmq clustered and load balanced on 3 nodes with 3600 nodes on each , having HA-TWO Queues policy.
Also we are bumping up from the RAM from 16G to 32G here is the new config , we are using [ {rabbitmq_stomp, [{tcp_listeners, [{"hosgtip", 6163}]}]},
{rabbit, [{reverse_dns_lookups, true},{vm_memory_high_watermark_paging_ratio, 0.7},{vm_memory_high_watermark, 0.7},{disk_free_limit, 1000000000},{cluster_partition_handling, autoheal}, {collect_statistics_interval, 30000}]},
{rabbitmq_management_agent, [{force_fine_statistics, false}]} ]. On Tue, Aug 26, 2014 at 12:01 PM, Michael Klishin <[hidden email]> wrote: On 26 August 2014 at 22:59:09, Jason McIntosh ([hidden email]) wrote: _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 26 August 2014 at 23:47:50, srikanth tns ([hidden email]) wrote:
> > what are the changes we see on the UI statistics ? I have applied > it on our test instances and see one of the main charts for message > rates is gone. Yes, message rate stats depend on fine grained statistics. This is mentioned in the docs. > Also we are bumping up from the RAM from 16G to 32G You can also bump the watermark ratio depending on what other software may be running on the same machine/VM. (but remember that OS and system services also need RAM, as do file system caches, so going above 0.85 is probably not a good idea) -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
Hi , Thanks for suggesting the changes. we have applied the relevant changes to config file for statistics and RAM upgrade. Things seem to be stable now but we will monitor the usage to see if it jumps off again.
here is a screenshot of how its looking now. Please let me know if this is promising , also is there any possibility of memory leaks and how we can avoid them?  Memory stats {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,4337647904}, {connection_procs,338878104},
{queue_procs,1193259768}, {plugins,328401480}, {other_proc,566879328},
{mnesia,331579864}, {mgmt_db,755366192}, {msg_index,22422152},
{other_ets,58232552}, {binary,402509464}, {code,17776116},
{atom,1621321}, {other_system,320721563}]}, {vm_memory_high_watermark,0.7},
{vm_memory_limit,23567893708}, {disk_free_limit,1000000000}, {disk_free,19428507648},
{file_descriptors, [{total_limit,327579}, {total_used,4289},
{sockets_limit,294819}, {sockets_used,4287}]}, {processes,[{limit,1048576},{used,197320}]},
{run_queue,0}, {uptime,4970}] {memory,
[{total,2662846904}, {connection_procs,188698640}, {queue_procs,929974408},
{plugins,265628672}, {other_proc,285984288}, {mnesia,333067736},
{mgmt_db,9096}, {msg_index,14039664}, {other_ets,34617944},
{binary,298184672}, {code,17779316}, {atom,1620513},
{other_system,293241955}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,23567893708},
{disk_free_limit,1000000000}, {disk_free,19331850240}, {file_descriptors,
[{total_limit,327579}, {total_used,3360}, {sockets_limit,294819},
{sockets_used,3358}]}, {processes,[{limit,1048576},{used,154555}]}, {run_queue,0},
{uptime,3812}] {memory, [{total,2548103256}, {connection_procs,216784920}, {queue_procs,846726064},
{plugins,247802288}, {other_proc,260433432}, {mnesia,333461048}, {mgmt_db,9096}, {msg_index,8830272}, {other_ets,33553920},
{binary,291250680}, {code,17776116}, {atom,1620513}, {other_system,289854907}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,23567893708},
{disk_free_limit,1000000000}, {disk_free,19382099968}, {file_descriptors, [{total_limit,327579}, {total_used,3112}, {sockets_limit,294819},
{sockets_used,3110}]}, {processes,[{limit,1048576},{used,143173}]}, {run_queue,0}, {uptime,3777}] On Tue, Aug 26, 2014 at 12:50 PM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 27 August 2014 at 04:13:33, srikanth tns ([hidden email]) wrote:
> > here is a screenshot of how its looking now. Please let me know > if this is promising , also is there any possibility of memory > leaks and how we can avoid them? Note that this is not necessarily a leak. You have over 100K queues, 100K consumers and you (used to) collect fine grained stats to display message rate, with a retention period in hours it's kind of expected that all those stats will consume quite a bit of RAM. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
So which retention policy is suggested ? global / basic / detailed ? setting it to coarse grained , make it global / basic ?
On Tue, Aug 26, 2014 at 5:19 PM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 27 August 2014 at 04:30:50, srikanth tns ([hidden email]) wrote:
> > So which retention policy is suggested ? global / basic / detailed > ? > setting it to coarse grained , make it global / basic ? There is no hard-and-fast rule but generally we can say that the more detailed you go (e.g. detailed), the shorter the period should be (I'd say 15 minutes is plenty for many cases). Global stats can be collected for longer periods, so that your overview page can be more helpful at detecting anomalies. Again, all of this depends on your team's needs but collecting detailed stats over the course of hours leads to RAM consumption that is hard to justify if you ask me. Experiment and see what works for you. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 27 August 2014 at 04:36:52, Michael Klishin ([hidden email]) wrote:
> > (I'd say 15 minutes > is plenty for many cases) This should read "15 seconds". -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
Cool, since we removed the fine grained statistics , we are off the detailed retention policy and running global / basic ? Is there any optimal value for global / basic policy ? Our needs are to just monitor the number of connections and use that data to load balance our connections.
On Tue, Aug 26, 2014 at 5:37 PM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 27 August 2014 at 04:44:04, srikanth tns ([hidden email]) wrote:
> Is there any optimal value for global / basic policy ? If you're happy with the current RAM use levels for stats DB, I'd recommend leaving things as they are for a while. > Our needs > are to just monitor the number of connections and use that data > to load balance our connections. If you don't use message rates info, you probably don't need fine grained stats. Again, if things work for you as they are after the change, I'd leave it this way until the need to re-enable fine grained stats arises. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
Great one of the things I saw when the rabbitmq was running high memory , the plugin value is -ve {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,9513029096}, {connection_procs,190083816},
{queue_procs,563064488}, {plugins,-4964607832}, {other_proc,5354348816},
{mnesia,289348840}, {mgmt_db,7437862072}, {msg_index,16722336},
{other_ets,50810936}, {binary,253822816}, {code,17776116},
{atom,1622937}, here is the list of plugin enabled [root@adsmv4218 ~]# rabbitmq-plugins list
[E] amqp_client 3.2.4 [ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d [ ] eldap 3.2.4-gite309de4 [e] mochiweb 2.7.0-rmq3.2.4-git680dba8
[ ] rabbitmq_amqp1_0 3.2.4 [ ] rabbitmq_auth_backend_ldap 3.2.4 [ ] rabbitmq_auth_mechanism_ssl 3.2.4 [ ] rabbitmq_consistent_hash_exchange 3.2.4
[ ] rabbitmq_federation 3.2.4 [ ] rabbitmq_federation_management 3.2.4 [ ] rabbitmq_jsonrpc 3.2.4 [ ] rabbitmq_jsonrpc_channel 3.2.4 [ ] rabbitmq_jsonrpc_channel_examples 3.2.4
[E] rabbitmq_management 3.2.4 [e] rabbitmq_management_agent 3.2.4 [ ] rabbitmq_management_visualiser 3.2.4 [ ] rabbitmq_mqtt 3.2.4
[ ] rabbitmq_shovel 3.2.4 [ ] rabbitmq_shovel_management 3.2.4 [E] rabbitmq_stomp 3.2.4 [ ] rabbitmq_tracing 3.2.4 [e] rabbitmq_web_dispatch 3.2.4
[ ] rabbitmq_web_stomp 3.2.4 [ ] rabbitmq_web_stomp_examples 3.2.4 [ ] rfc4627_jsonrpc 3.2.4-git5e67120 [ ] sockjs 0.3.4-rmq3.2.4-git3132eb9
[e] webmachine 1.10.3-rmq3.2.4-gite9359c7 On Tue, Aug 26, 2014 at 5:46 PM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
On 27 August 2014 at 04:56:04, srikanth tns ([hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high > memory , the plugin value is -ve Do you use STOMP actively? We've improved some things in STOMP and MQTT plugins in the upcoming 3.4.0 release that would lead to high RAM usage reported by the plugins (and lower by binaries). This is not a leak but something that skews the picture quite a bit and results in higher RAM use from "non-core" protocol connections for no good reason. -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
In reply to this post by srikanth tns
On 27 August 2014 at 04:56:04, srikanth tns ([hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high > memory , the plugin value is -ve And the negative value is indeed a bug, 26058, which was fixed in 3.3.0: http://www.rabbitmq.com/release-notes/README-3.3.0.txt -- MK Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
Re: Rabbitmq 3.2.4 running out of memory
|
|
Yes we use STOMP actively to send message to the targets hosts via mcollective rpc. The messages are sent to the targets every 10 secs. Does it cause any issue ? On Tue, Aug 26, 2014 at 6:01 PM, Michael Klishin <[hidden email]> wrote:
_______________________________________________ rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users, please subscribe to the new list! [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss |
«
Return to RabbitMQ
|
1 view|%1 views
| Free forum by Nabble | Edit this page |