|
|
12
|
|
Hi
Can someone please let us know wht we need to do ?
we are getting out of memory errors in the rabbitmq 3.2.4 when we are trying to access rabbitmq on port 15672 using connections api . The script calls curl via shell console.
curl -i -s --connect-timeout 5 -u admin:changeme http://$hostname1:15672/api/connections?columns=node,name
This curl call is made from many hosts at same to get the number of connections running on rabbitmq and we see the connections are blocked on this port.
netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c
3 CLOSE_WAIT 1882 ESTABLISHED 1 LISTEN 10 TIME_WAIT
Thanks Srikanth
Logs show this:
vm_memory_high_watermark set. Memory used:14888001288 allowed:11753753804
=WARNING REPORT==== 26-Aug-2014::05:58:18 === memory resource limit alarm set on node rabbit@adsmv0019.
********************************************************** *** Publishers will be blocked until this alarm clears ***
**********************************************************
=INFO REPORT==== 26-Aug-2014::05:58:20 === vm_memory_high_watermark clear. Memory used:8686452200 allowed:11753753804
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 26 August 2014 at 20:02:09, srikanth tns ( [hidden email]) wrote:
> > Logs show this:
>
> vm_memory_high_watermark set. Memory used:14888001288
> allowed:11753753804
>
> =WARNING REPORT==== 26-Aug-2014::05:58:18
> ===
> memory resource limit alarm set on node rabbit@adsmv0019.
>
> **********************************************************
> *** Publishers will be blocked until this alarm clears ***
> **********************************************************
>
>
> =INFO REPORT==== 26-Aug-2014::05:58:20
> ===
> vm_memory_high_watermark clear. Memory used:8686452200
> allowed:11753753804
Use rabbitmqctl report or management UI (node info page) to find out what takes memory.
Then either eliminate that thing (likely it is messages in the queue,
there is a fixed RAM cost per message: the queue index entry(-ies))
or bump the watermark.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 26 August 2014 at 20:02:09, srikanth tns ( [hidden email]) wrote:
> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c
> 3 CLOSE_WAIT
> 1882 ESTABLISHED
> 1 LISTEN
> 10 TIME_WAIT
That's about 1900 concurrent connections to a single node. Without knowing
how many channels can be open on the open ones, that's still a fairly
large number for a single node.
Consider clustering this node with a couple more and spread the connections
between them, e.g. using an HAproxy instance in front of the cluster.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
The UI is not accessible , The 3-nodes are clustered with HA two policy
rabbitmqctl reports shows
Reporting server status on {{2014,8,26},{18,39,31}}
... Status of node rabbit@adsmv0020 ... [{pid,22153}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},
{rabbitmq_management,"RabbitMQ Management Console","3.2.4"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"},
{os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"}, {sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,2285543720}, {connection_procs,207275088}, {queue_procs,599282104}, {plugins,234739776}, {other_proc,309911344},
{mnesia,289075584}, {mgmt_db,9096}, {msg_index,14688048}, {other_ets,28236624}, {binary,290512400}, {code,17776116}, {atom,1621321},
{other_system,292416219}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19386892288}, {file_descriptors,
[{total_limit,327579}, {total_used,3137}, {sockets_limit,294819}, {sockets_used,3135}]}, {processes,[{limit,1048576},{used,144321}]}, {run_queue,0},
{uptime,6267}]
Cluster status of node rabbit@adsmv0020 ... [{nodes,[{disc,[rabbit@adsmv0019,rabbit@adsmv0018]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0018,rabbit@adsmv0019,rabbit@adsmv0020]},
{partitions,[]}]
Application environment of node rabbit@adsmv0020 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,
rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,
sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array,
rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0020-plugins-expand"},
{queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true},
{server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary,
{packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}},
{exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}]
Status of node rabbit@adsmv0019 ... [{pid,27139}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"}, {os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},
{xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"},
{sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,2375638736},
{connection_procs,228104952}, {queue_procs,643238504}, {plugins,259036872}, {other_proc,276878920}, {mnesia,288547136}, {mgmt_db,9096},
{msg_index,14503600}, {other_ets,30000664}, {binary,321298960}, {binary,321298960}, {code,17776116}, {atom,1621321},
{other_system,294622595}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19439706112}, {file_descriptors,
[{total_limit,327579}, {total_used,3466}, {sockets_limit,294819}, {sockets_used,3464}]}, {processes,[{limit,1048576},{used,159459}]}, {run_queue,0},
{uptime,6302}]
Cluster status of node rabbit@adsmv0019 ... [{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0018,rabbit@adsmv0019]},
{partitions,[]}]
Application environment of node rabbit@adsmv0019 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue,
rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue,
rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,
file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]},
{log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},
{plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0019-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true},
{server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary,
{packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}},
{exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}]
Status of node rabbit@adsmv0018 ... [{pid,20727}, {running_applications, [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"}, {rabbit,"RabbitMQ","3.2.4"}, {os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},
{xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"},
{sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,6095435856},
{connection_procs,185360136}, {queue_procs,526782896}, {plugins,-1872186488}, {other_proc,2447546120}, {mnesia,289695904}, {mgmt_db,3870226464},
{msg_index,23253920}, {other_ets,51366128}, {binary,249989160}, {code,17776116}, {atom,1622129}, {other_system,304003371}]},
{vm_memory_high_watermark,0.7}, {vm_memory_limit,11753753804}, {disk_free_limit,1000000000}, {disk_free,19338166272}, {file_descriptors, [{total_limit,327579},
{total_used,2748}, {sockets_limit,294819}, {sockets_used,2746}]}, {processes,[{limit,1048576},{used,126430}]}, {run_queue,0}, {uptime,6789}]
Cluster status of node rabbit@adsmv0018 ... [{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]}, {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0019,rabbit@adsmv0018]},
{partitions,[]}]
Application environment of node rabbit@adsmv0018 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,autoheal}, {collect_statistics,fine}, {collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]},
{default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,
rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,
sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array,
rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0018-plugins-expand"},
{queue_index_max_journal_entries,65536}, {reverse_dns_lookups,true}, {server_properties,[]},
{ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw},
{reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]},
{tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.7}, {vm_memory_high_watermark_paging_ratio,0.7}]
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
Looking at your memory: {memory, [{total,<a href="tel:6095435856" value="+16095435856" target="_blank">6095435856},
{connection_procs,185360136}, 185MB {queue_procs,526782896}, 526MB
{plugins,-1872186488}, {other_proc,2447546120}, 2447MB {mnesia,289695904}, 290MB
{mgmt_db,3870226464},3870MB {msg_index,23253920},23MB
{other_ets,51366128},51MB {binary,249989160},250MB {code, 17776116},17MB
{atom,1622129}, 1MB {other_system,304003371}304MB
It appears your available memory is about 12GB available, .07 * of that would mean about 8GB available, and you're using right at that from all I'm seeing. I could be mis-understanding a parameter. But it looks like you have a LOT of queues, and then a LOT of statistics on those queues - perhaps reducing the statistics gathering rate or some other tuning might help?
Jason _______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
what are the changes we see on the UI statistics ? I have applied it on our test instances and see one of the main charts for message rates is gone. Currently we are servicing 10800 nodes on rabbitmq clustered and load balanced on 3 nodes with 3600 nodes on each , having HA-TWO Queues policy.
Also we are bumping up from the RAM from 16G to 32G
here is the new config , we are using
[ {rabbitmq_stomp, [{tcp_listeners, [{"hosgtip", 6163}]}]},
{rabbit, [{reverse_dns_lookups, true},{vm_memory_high_watermark_paging_ratio, 0.7},{vm_memory_high_watermark, 0.7},{disk_free_limit, 1000000000},{cluster_partition_handling, autoheal}, {collect_statistics_interval, 30000}]},
{rabbitmq_management_agent, [{force_fine_statistics, false}]} ].
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 26 August 2014 at 23:47:50, srikanth tns ( [hidden email]) wrote:
> > what are the changes we see on the UI statistics ? I have applied
> it on our test instances and see one of the main charts for message
> rates is gone.
Yes, message rate stats depend on fine grained statistics. This is mentioned
in the docs.
> Also we are bumping up from the RAM from 16G to 32G
You can also bump the watermark ratio depending on what other software
may be running on the same machine/VM.
(but remember that OS and system services
also need RAM, as do file system caches, so going above 0.85 is probably
not a good idea)
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
Hi ,
Thanks for suggesting the changes. we have applied the relevant changes to config file for statistics and RAM upgrade. Things seem to be stable now but we will monitor the usage to see if it jumps off again.
here is a screenshot of how its looking now. Please let me know if this is promising , also is there any possibility of memory leaks and how we can avoid them?
Memory stats
{erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,4337647904}, {connection_procs,338878104},
{queue_procs,1193259768}, {plugins,328401480}, {other_proc,566879328},
{mnesia,331579864}, {mgmt_db,755366192}, {msg_index,22422152},
{other_ets,58232552}, {binary,402509464}, {code,17776116},
{atom,1621321}, {other_system,320721563}]}, {vm_memory_high_watermark,0.7},
{vm_memory_limit,23567893708}, {disk_free_limit,1000000000}, {disk_free,19428507648},
{file_descriptors, [{total_limit,327579}, {total_used,4289},
{sockets_limit,294819}, {sockets_used,4287}]}, {processes,[{limit,1048576},{used,197320}]},
{run_queue,0}, {uptime,4970}]
{memory,
[{total,2662846904}, {connection_procs,188698640}, {queue_procs,929974408},
{plugins,265628672}, {other_proc,285984288}, {mnesia,333067736},
{mgmt_db,9096}, {msg_index,14039664}, {other_ets,34617944},
{binary,298184672}, {code,17779316}, {atom,1620513},
{other_system,293241955}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,23567893708},
{disk_free_limit,1000000000}, {disk_free,19331850240}, {file_descriptors,
[{total_limit,327579}, {total_used,3360}, {sockets_limit,294819},
{sockets_used,3358}]}, {processes,[{limit,1048576},{used,154555}]}, {run_queue,0},
{uptime,3812}]
{memory, [{total,2548103256}, {connection_procs,216784920}, {queue_procs,846726064},
{plugins,247802288}, {other_proc,260433432}, {mnesia,333461048}, {mgmt_db,9096}, {msg_index,8830272}, {other_ets,33553920},
{binary,291250680}, {code,17776116}, {atom,1620513}, {other_system,289854907}]}, {vm_memory_high_watermark,0.7}, {vm_memory_limit,23567893708},
{disk_free_limit,1000000000}, {disk_free,19382099968}, {file_descriptors, [{total_limit,327579}, {total_used,3112}, {sockets_limit,294819},
{sockets_used,3110}]}, {processes,[{limit,1048576},{used,143173}]}, {run_queue,0}, {uptime,3777}] _______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 27 August 2014 at 04:13:33, srikanth tns ( [hidden email]) wrote:
> > here is a screenshot of how its looking now. Please let me know
> if this is promising , also is there any possibility of memory
> leaks and how we can avoid them?
Note that this is not necessarily a leak.
You have over 100K queues,
100K consumers and you (used to) collect fine grained
stats to display message rate, with a retention period in hours it's kind of
expected that all those stats will consume quite a bit of RAM.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 27 August 2014 at 04:30:50, srikanth tns ( [hidden email]) wrote:
> > So which retention policy is suggested ? global / basic / detailed
> ?
> setting it to coarse grained , make it global / basic ?
There is no hard-and-fast rule but generally we can say that the more detailed
you go (e.g. detailed), the shorter the period should be (I'd say 15 minutes
is plenty for many cases). Global stats can be collected for longer periods,
so that your overview page can be more helpful at detecting anomalies.
Again, all of this depends on your team's needs but collecting detailed stats
over the course of hours leads to RAM consumption that is hard to justify
if you ask me.
Experiment and see what works for you.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 27 August 2014 at 04:44:04, srikanth tns ( [hidden email]) wrote:
> Is there any optimal value for global / basic policy ?
If you're happy with the current RAM use levels for stats DB, I'd recommend
leaving things as they are for a while.
> Our needs
> are to just monitor the number of connections and use that data
> to load balance our connections.
If you don't use message rates info, you probably don't need fine grained stats.
Again, if things work for you as they are after the change, I'd leave it
this way until the need to re-enable fine grained stats arises.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
Great one of the things I saw when the rabbitmq was running high memory , the plugin value is -ve
{erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory, [{total,9513029096}, {connection_procs,190083816},
{queue_procs,563064488}, {plugins,-4964607832}, {other_proc,5354348816},
{mnesia,289348840}, {mgmt_db,7437862072}, {msg_index,16722336},
{other_ets,50810936}, {binary,253822816}, {code,17776116},
{atom,1622937},
here is the list of plugin enabled
[root@adsmv4218 ~]# rabbitmq-plugins list
[E] amqp_client 3.2.4 [ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d [ ] eldap 3.2.4-gite309de4 [e] mochiweb 2.7.0-rmq3.2.4-git680dba8
[ ] rabbitmq_amqp1_0 3.2.4 [ ] rabbitmq_auth_backend_ldap 3.2.4 [ ] rabbitmq_auth_mechanism_ssl 3.2.4 [ ] rabbitmq_consistent_hash_exchange 3.2.4
[ ] rabbitmq_federation 3.2.4 [ ] rabbitmq_federation_management 3.2.4 [ ] rabbitmq_jsonrpc 3.2.4 [ ] rabbitmq_jsonrpc_channel 3.2.4 [ ] rabbitmq_jsonrpc_channel_examples 3.2.4
[E] rabbitmq_management 3.2.4 [e] rabbitmq_management_agent 3.2.4 [ ] rabbitmq_management_visualiser 3.2.4 [ ] rabbitmq_mqtt 3.2.4
[ ] rabbitmq_shovel 3.2.4 [ ] rabbitmq_shovel_management 3.2.4 [E] rabbitmq_stomp 3.2.4 [ ] rabbitmq_tracing 3.2.4 [e] rabbitmq_web_dispatch 3.2.4
[ ] rabbitmq_web_stomp 3.2.4 [ ] rabbitmq_web_stomp_examples 3.2.4 [ ] rfc4627_jsonrpc 3.2.4-git5e67120 [ ] sockjs 0.3.4-rmq3.2.4-git3132eb9
[e] webmachine 1.10.3-rmq3.2.4-gite9359c7 _______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
|
|
On 27 August 2014 at 04:56:04, srikanth tns ( [hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high
> memory , the plugin value is -ve
Do you use STOMP actively?
We've improved some things in STOMP and MQTT plugins in the upcoming
3.4.0 release that would lead to high RAM usage reported by the plugins
(and lower by binaries). This is not a leak but something that skews
the picture quite a bit and results in higher RAM use from "non-core" protocol
connections for no good reason.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
|
12
|
Locked
