Rabbitmq 3.2.4 running out of memory

[srikanth tns]

Hi 

Can someone please let us know wht we need to do ? 

we are getting out of memory errors in the rabbitmq 3.2.4 when we are trying to access rabbitmq on port 15672 using connections api . The script calls curl via shell console. 

curl -i -s --connect-timeout 5 -u admin:changeme http://$hostname1:15672/api/connections?columns=node,name

This curl call is made from many hosts at same to get the number of connections running on rabbitmq and we see the connections are blocked on this port.

netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c

      3 CLOSE_WAIT

   1882 ESTABLISHED

      1 LISTEN

     10 TIME_WAIT

Thanks

Srikanth

Logs show this:

vm_memory_high_watermark set. Memory used:14888001288  allowed:11753753804

=WARNING REPORT==== 26-Aug-2014::05:58:18  ===

memory resource limit alarm set on node  rabbit@adsmv0019.

**********************************************************

***  Publishers will be blocked until this alarm clears  ***

**********************************************************

=INFO REPORT====  26-Aug-2014::05:58:20 ===

vm_memory_high_watermark clear. Memory  used:8686452200 allowed:11753753804

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:

> > Logs show this:
>  
> vm_memory_high_watermark set. Memory used:14888001288
> allowed:11753753804
>  
> =WARNING REPORT==== 26-Aug-2014::05:58:18  
> ===
> memory resource limit alarm set on node rabbit@adsmv0019.
>  
> **********************************************************  
> *** Publishers will be blocked until this alarm clears ***
> **********************************************************  
>  
>  
> =INFO REPORT==== 26-Aug-2014::05:58:20  
> ===
> vm_memory_high_watermark clear. Memory used:8686452200
> allowed:11753753804

Use rabbitmqctl report or management UI (node info page)  to find out what takes memory.

Then either eliminate that thing (likely it is messages in the queue,
there is a fixed RAM cost per message: the queue index entry(-ies))
or bump the watermark.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:
> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c  
> 3 CLOSE_WAIT
> 1882 ESTABLISHED
> 1 LISTEN
> 10 TIME_WAIT

That's about 1900 concurrent connections to a single node. Without knowing
how many channels can be open on the open ones, that's still a fairly
large number for a single node.

Consider clustering this node with a couple more and spread the connections
between them, e.g. using an HAproxy instance in front of the cluster. 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:
> > Thanks
> Srikanth

Also: this list has moved to rabbitmq-users:
https://groups.google.com/forum/#!forum/rabbitmq-users 

Please start all new discussions there.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

The UI is not accessible , The 3-nodes are clustered with HA two policy

rabbitmqctl reports shows 

Reporting server status on {{2014,8,26},{18,39,31}}

 ...

Status of node rabbit@adsmv0020 ...

[{pid,22153},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,2285543720},

      {connection_procs,207275088},

      {queue_procs,599282104},

      {plugins,234739776},

      {other_proc,309911344},

      {mnesia,289075584},

      {mgmt_db,9096},

      {msg_index,14688048},

      {other_ets,28236624},

      {binary,290512400},

      {code,17776116},

      {atom,1621321},

      {other_system,292416219}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,19386892288},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,3137},

      {sockets_limit,294819},

      {sockets_used,3135}]},

 {processes,[{limit,1048576},{used,144321}]},

 {run_queue,0},

 {uptime,6267}]

Cluster status of node rabbit@adsmv0020 ...

[{nodes,[{disc,[rabbit@adsmv0019,rabbit@adsmv0018]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0018,rabbit@adsmv0019,rabbit@adsmv0020]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0020 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0020-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

Status of node rabbit@adsmv0019 ...

[{pid,27139},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,2375638736},

      {connection_procs,228104952},

      {queue_procs,643238504},

      {plugins,259036872},

      {other_proc,276878920},

      {mnesia,288547136},

      {mgmt_db,9096},

      {msg_index,14503600},

      {other_ets,30000664},

      {binary,321298960},

      {binary,321298960},

      {code,17776116},

      {atom,1621321},

      {other_system,294622595}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,19439706112},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,3466},

      {sockets_limit,294819},

      {sockets_used,3464}]},

 {processes,[{limit,1048576},{used,159459}]},

 {run_queue,0},

 {uptime,6302}]

Cluster status of node rabbit@adsmv0019 ...

[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0018,rabbit@adsmv0019]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0019 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0019-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

Status of node rabbit@adsmv0018 ...

[{pid,20727},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,6095435856},

      {connection_procs,185360136},

      {queue_procs,526782896},

      {plugins,-1872186488},

      {other_proc,2447546120},

      {mnesia,289695904},

      {mgmt_db,3870226464},

      {msg_index,23253920},

      {other_ets,51366128},

      {binary,249989160},

      {code,17776116},

      {atom,1622129},

      {other_system,304003371}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,19338166272},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,2748},

      {sockets_limit,294819},

      {sockets_used,2746}]},

 {processes,[{limit,1048576},{used,126430}]},

 {run_queue,0},

 {uptime,6789}]

Cluster status of node rabbit@adsmv0018 ...

[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0019,rabbit@adsmv0018]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0018 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0018-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

On Tue, Aug 26, 2014 at 11:42 AM, Michael Klishin <[hidden email]> wrote:

On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:

> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c
> 3 CLOSE_WAIT
> 1882 ESTABLISHED
> 1 LISTEN
> 10 TIME_WAIT

That's about 1900 concurrent connections to a single node. Without knowing
how many channels can be open on the open ones, that's still a fairly
large number for a single node.

Consider clustering this node with a couple more and spread the connections
between them, e.g. using an HAproxy instance in front of the cluster. 

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[mc717990]

Looking at your memory:

 {memory,

     [{total,<a href="tel:[phone]" value="[phone]" target="_blank">6095435856},

      {connection_procs,185360136}, 185MB

      {queue_procs,526782896}, 526MB

      {plugins,-1872186488},

      {other_proc,2447546120}, 2447MB

      {mnesia,289695904}, 290MB

      {mgmt_db,3870226464},3870MB

      {msg_index,23253920},23MB

      {other_ets,51366128},51MB

      {binary,249989160},250MB

      {code,   17776116},17MB

      {atom,1622129}, 1MB

      {other_system,304003371}304MB

It appears your available memory is about 12GB available, .07 * of that would mean about 8GB available, and you're using right at that from all I'm seeing.  I could be mis-understanding a parameter.  But it looks like you have a LOT of queues, and then a LOT of statistics on those queues - perhaps reducing the statistics gathering rate or some other tuning might help?

Jason

On Tue, Aug 26, 2014 at 1:47 PM, srikanth tns <[hidden email]> wrote:

The UI is not accessible , The 3-nodes are clustered with HA two policy

rabbitmqctl reports shows 

Reporting server status on {{2014,8,26},{18,39,31}}

 ...

Status of node rabbit@adsmv0020 ...

[{pid,22153},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,<a href="tel:[phone]" value="[phone]" target="_blank">2285543720},

      {connection_procs,207275088},

      {queue_procs,599282104},

      {plugins,234739776},

      {other_proc,309911344},

      {mnesia,289075584},

      {mgmt_db,9096},

      {msg_index,14688048},

      {other_ets,28236624},

      {binary,290512400},

      {code,17776116},

      {atom,1621321},

      {other_system,292416219}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,<a href="tel:[phone]" value="[phone]" target="_blank">19386892288},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,3137},

      {sockets_limit,294819},

      {sockets_used,3135}]},

 {processes,[{limit,1048576},{used,144321}]},

 {run_queue,0},

 {uptime,6267}]

Cluster status of node rabbit@adsmv0020 ...

[{nodes,[{disc,[rabbit@adsmv0019,rabbit@adsmv0018]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0018,rabbit@adsmv0019,rabbit@adsmv0020]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0020 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0020-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

Status of node rabbit@adsmv0019 ...

[{pid,27139},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,2375638736},

      {connection_procs,228104952},

      {queue_procs,643238504},

      {plugins,259036872},

      {other_proc,276878920},

      {mnesia,288547136},

      {mgmt_db,9096},

      {msg_index,14503600},

      {other_ets,30000664},

      {binary,321298960},

      {binary,321298960},

      {code,17776116},

      {atom,1621321},

      {other_system,294622595}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,19439706112},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,3466},

      {sockets_limit,294819},

      {sockets_used,3464}]},

 {processes,[{limit,1048576},{used,159459}]},

 {run_queue,0},

 {uptime,6302}]

Cluster status of node rabbit@adsmv0019 ...

[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0018,rabbit@adsmv0019]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0019 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0019-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

Status of node rabbit@adsmv0018 ...

[{pid,20727},

 {running_applications,

     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},

      {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},

      {rabbit,"RabbitMQ","3.2.4"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},

      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,<a href="tel:[phone]" value="[phone]" target="_blank">6095435856},

      {connection_procs,185360136},

      {queue_procs,526782896},

      {plugins,-1872186488},

      {other_proc,2447546120},

      {mnesia,289695904},

      {mgmt_db,3870226464},

      {msg_index,23253920},

      {other_ets,51366128},

      {binary,249989160},

      {code,17776116},

      {atom,1622129},

      {other_system,304003371}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,11753753804},

 {disk_free_limit,1000000000},

 {disk_free,19338166272},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,2748},

      {sockets_limit,294819},

      {sockets_used,2746}]},

 {processes,[{limit,1048576},{used,126430}]},

 {run_queue,0},

 {uptime,6789}]

Cluster status of node rabbit@adsmv0018 ...

[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},

 {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0019,rabbit@adsmv0018]},

 {partitions,[]}]

Application environment of node rabbit@adsmv0018 ...

[{auth_backends,[rabbit_auth_backend_internal]},

 {auth_mechanisms,['PLAIN','AMQPLAIN']},

 {backing_queue_module,rabbit_variable_queue},

 {cluster_nodes,{[],disc}},

 {cluster_partition_handling,autoheal},

 {collect_statistics,fine},

 {collect_statistics_interval,5000},

 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

 {default_user,<<"guest">>},

 {default_user_tags,[administrator]},

 {default_vhost,<<"/">>},

 {delegate_count,16},

 {disk_free_limit,1000000000},

 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},

 {error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {frame_max,131072},

 {halt_on_upgrade_failure,true},

 {heartbeat,580},

 {hipe_compile,false},

 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,

                rabbit_basic,rabbit_event,lists,queue,priority_queue,

                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,

                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,

                rabbit_amqqueue_process,rabbit_variable_queue,

                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,

                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,

                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,

                file_handle_cache,rabbit_msg_store,array,

                rabbit_msg_store_ets_index,rabbit_msg_file,

                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,

                ssl]},

 {included_applications,[]},

 {log_levels,[{connection,info}]},

 {msg_store_file_size_limit,16777216},

 {msg_store_index_module,rabbit_msg_store_ets_index},

 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},

 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0018-plugins-expand"},

 {queue_index_max_journal_entries,65536},

 {reverse_dns_lookups,true},

 {sasl_error_logger,{file,"/var/log/rabbitmq/[hidden email]"}},

 {server_properties,[]},

 {ssl_apps,[asn1,crypto,public_key,ssl]},

 {ssl_cert_login_from,distinguished_name},

 {ssl_listeners,[]},

 {ssl_options,[]},

 {tcp_listen_options,[binary,

                      {packet,raw},

                      {reuseaddr,true},

                      {backlog,128},

                      {nodelay,true},

                      {linger,{true,0}},

                      {exit_on_close,false}]},

 {tcp_listeners,[5672]},

 {trace_vhosts,[]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_high_watermark_paging_ratio,0.7}]

On Tue, Aug 26, 2014 at 11:42 AM, Michael Klishin <[hidden email]> wrote:

On 26 August 2014 at 20:02:09, srikanth tns ([hidden email]) wrote:

> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c
> 3 CLOSE_WAIT
> 1882 ESTABLISHED
> 1 LISTEN
> 10 TIME_WAIT

That's about 1900 concurrent connections to a single node. Without knowing
how many channels can be open on the open ones, that's still a fairly
large number for a single node.

Consider clustering this node with a couple more and spread the connections
between them, e.g. using an HAproxy instance in front of the cluster. 

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

--

Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 26 August 2014 at 22:59:09, Jason McIntosh ([hidden email]) wrote:
> {mgmt_db,3870226464},3870MB

Thanks, Jason.  So here's our key offender.

Besides memory leaks that are fixed in the statistics DB from time to time,
you can turn off fine grained stats. This should drastically reduce the
amount of RAM used by the stats database.

See http://rabbitmq.com/management.html.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

what are the changes we see on the UI statistics ? I have applied it on our test instances and see one of the main charts for message rates is gone.

Currently we are servicing 10800 nodes on rabbitmq clustered and load balanced on 3 nodes with 3600 nodes on each , having HA-TWO Queues policy.

Also we are bumping up from the RAM from 16G to 32G

here is the new config , we are using 

[

 {rabbitmq_stomp, [{tcp_listeners, [{"hosgtip", 6163}]}]},

 {rabbit, [{reverse_dns_lookups, true},{vm_memory_high_watermark_paging_ratio, 0.7},{vm_memory_high_watermark, 0.7},{disk_free_limit, 1000000000},{cluster_partition_handling, autoheal},

  {collect_statistics_interval, 30000}]},

 {rabbitmq_management_agent, [{force_fine_statistics, false}]}

].

On Tue, Aug 26, 2014 at 12:01 PM, Michael Klishin <[hidden email]> wrote:

On 26 August 2014 at 22:59:09, Jason McIntosh ([hidden email]) wrote:
> {mgmt_db,3870226464},3870MB

Thanks, Jason.  So here's our key offender.

Besides memory leaks that are fixed in the statistics DB from time to time,
you can turn off fine grained stats. This should drastically reduce the
amount of RAM used by the stats database.

See http://rabbitmq.com/management.html.

--
MK

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

 On 26 August 2014 at 23:47:50, srikanth tns ([hidden email]) wrote:
> > what are the changes we see on the UI statistics ? I have applied  
> it on our test instances and see one of the main charts for message  
> rates is gone.

Yes, message rate stats depend on fine grained statistics. This is mentioned
in the docs.

> Also we are bumping up from the RAM from 16G to 32G

You can also bump the watermark ratio depending on what other software
may be running on the same machine/VM.
(but remember that OS and system services
also need RAM, as do file system caches, so going above 0.85 is probably
not a good idea)
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

Hi , 

Thanks for suggesting the changes. 

we have applied the relevant changes to config file for statistics and RAM upgrade. Things seem to be stable now but we will monitor the usage to see if it jumps off again.

here is a screenshot of how its looking now. Please let me know if this is promising , also is there any possibility of memory leaks and how we can avoid them?

Memory stats

{erlang_version,

    "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

{memory,

    [{total,4337647904},

     {connection_procs,338878104},

     {queue_procs,1193259768},

     {plugins,328401480},

     {other_proc,566879328},

     {mnesia,331579864},

     {mgmt_db,755366192},

     {msg_index,22422152},

     {other_ets,58232552},

     {binary,402509464},

     {code,17776116},

     {atom,1621321},

     {other_system,320721563}]},

{vm_memory_high_watermark,0.7},

{vm_memory_limit,23567893708},

{disk_free_limit,1000000000},

{disk_free,19428507648},

{file_descriptors,

    [{total_limit,327579},

     {total_used,4289},

     {sockets_limit,294819},

     {sockets_used,4287}]},

{processes,[{limit,1048576},{used,197320}]},

{run_queue,0},

{uptime,4970}]

{memory,

    [{total,2662846904},

     {connection_procs,188698640},

     {queue_procs,929974408},

     {plugins,265628672},

     {other_proc,285984288},

     {mnesia,333067736},

     {mgmt_db,9096},

     {msg_index,14039664},

     {other_ets,34617944},

     {binary,298184672},

     {code,17779316},

     {atom,1620513},

     {other_system,293241955}]},

{vm_memory_high_watermark,0.7},

{vm_memory_limit,23567893708},

{disk_free_limit,1000000000},

{disk_free,19331850240},

{file_descriptors,

    [{total_limit,327579},

     {total_used,3360},

     {sockets_limit,294819},

     {sockets_used,3358}]},

{processes,[{limit,1048576},{used,154555}]},

{run_queue,0},

{uptime,3812}]

 {memory,

     [{total,2548103256},

      {connection_procs,216784920},

      {queue_procs,846726064},

      {plugins,247802288},

      {other_proc,260433432},

      {mnesia,333461048},

      {mgmt_db,9096},

      {msg_index,8830272},

      {other_ets,33553920},

      {binary,291250680},

      {code,17776116},

      {atom,1620513},

      {other_system,289854907}]},

 {vm_memory_high_watermark,0.7},

 {vm_memory_limit,23567893708},

 {disk_free_limit,1000000000},

 {disk_free,19382099968},

 {file_descriptors,

     [{total_limit,327579},

      {total_used,3112},

      {sockets_limit,294819},

      {sockets_used,3110}]},

 {processes,[{limit,1048576},{used,143173}]},

 {run_queue,0},

 {uptime,3777}]  

On Tue, Aug 26, 2014 at 12:50 PM, Michael Klishin <[hidden email]> wrote:

 On 26 August 2014 at 23:47:50, srikanth tns ([hidden email]) wrote:
> > what are the changes we see on the UI statistics ? I have applied
> it on our test instances and see one of the main charts for message
> rates is gone.

Yes, message rate stats depend on fine grained statistics. This is mentioned
in the docs.

> Also we are bumping up from the RAM from 16G to 32G

You can also bump the watermark ratio depending on what other software
may be running on the same machine/VM.
(but remember that OS and system services
also need RAM, as do file system caches, so going above 0.85 is probably
not a good idea)

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 27 August 2014 at 04:13:33, srikanth tns ([hidden email]) wrote:
> > here is a screenshot of how its looking now. Please let me know  
> if this is promising , also is there any possibility of memory  
> leaks and how we can avoid them?

Note that this is not necessarily a leak.

You have over 100K queues,
100K consumers and you (used to) collect fine grained
stats to display message rate, with a retention period in hours it's kind of
expected that all those stats will consume quite a bit of RAM. 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

So which retention policy is suggested ? global / basic / detailed ?

setting it to coarse grained , make it global / basic ? 

On Tue, Aug 26, 2014 at 5:19 PM, Michael Klishin <[hidden email]> wrote:

On 27 August 2014 at 04:13:33, srikanth tns ([hidden email]) wrote:
> > here is a screenshot of how its looking now. Please let me know
> if this is promising , also is there any possibility of memory
> leaks and how we can avoid them?

Note that this is not necessarily a leak.

You have over 100K queues,
100K consumers and you (used to) collect fine grained
stats to display message rate, with a retention period in hours it's kind of
expected that all those stats will consume quite a bit of RAM. 

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 27 August 2014 at 04:30:50, srikanth tns ([hidden email]) wrote:
> > So which retention policy is suggested ? global / basic / detailed
> ?
> setting it to coarse grained , make it global / basic ?

There is no hard-and-fast rule but generally we can say that the more detailed
you go (e.g. detailed), the shorter the period should be (I'd say 15 minutes
is plenty for many cases). Global stats can be collected for longer periods,
so that your overview page can be more helpful at detecting anomalies.

Again, all of this depends on your team's needs but collecting detailed stats
over the course of hours leads to RAM consumption that is hard to justify
if you ask me.

Experiment and see what works for you.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 27 August 2014 at 04:36:52, Michael Klishin ([hidden email]) wrote:
> > (I'd say 15 minutes
> is plenty for many cases)

This should read "15 seconds". 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

Cool, since we removed the fine grained statistics , we are off the detailed retention policy and running global / basic ? 

Is there any optimal value for global / basic policy ? Our needs are to just monitor the number of connections and use that data to load balance our connections.

On Tue, Aug 26, 2014 at 5:37 PM, Michael Klishin <[hidden email]> wrote:

On 27 August 2014 at 04:36:52, Michael Klishin ([hidden email]) wrote:
> > (I'd say 15 minutes
> is plenty for many cases)

This should read "15 seconds". 

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

 On 27 August 2014 at 04:44:04, srikanth tns ([hidden email]) wrote:
> Is there any optimal value for global / basic policy ?

If you're happy with the current RAM use levels for stats DB, I'd recommend
leaving things as they are for a while.

> Our needs  
> are to just monitor the number of connections and use that data  
> to load balance our connections.

If you don't use message rates info, you probably don't need fine grained stats.
Again, if things work for you as they are after the change, I'd leave it
this way until the need to re-enable fine grained stats arises.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

Great one of the things I saw when the rabbitmq was running high memory , the plugin value is -ve 

{erlang_version,

    "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},

{memory,

    [{total,9513029096},

     {connection_procs,190083816},

     {queue_procs,563064488},

     {plugins,-4964607832},

     {other_proc,5354348816},

     {mnesia,289348840},

     {mgmt_db,7437862072},

     {msg_index,16722336},

     {other_ets,50810936},

     {binary,253822816},

     {code,17776116},

     {atom,1622937},

here is the list of plugin enabled 

[root@adsmv4218 ~]# rabbitmq-plugins list

[E] amqp_client                       3.2.4

[ ] cowboy                            0.5.0-rmq3.2.4-git4b93c2d

[ ] eldap                             3.2.4-gite309de4

[e] mochiweb                          2.7.0-rmq3.2.4-git680dba8

[ ] rabbitmq_amqp1_0                  3.2.4

[ ] rabbitmq_auth_backend_ldap        3.2.4

[ ] rabbitmq_auth_mechanism_ssl       3.2.4

[ ] rabbitmq_consistent_hash_exchange 3.2.4

[ ] rabbitmq_federation               3.2.4

[ ] rabbitmq_federation_management    3.2.4

[ ] rabbitmq_jsonrpc                  3.2.4

[ ] rabbitmq_jsonrpc_channel          3.2.4

[ ] rabbitmq_jsonrpc_channel_examples 3.2.4

[E] rabbitmq_management               3.2.4

[e] rabbitmq_management_agent         3.2.4

[ ] rabbitmq_management_visualiser    3.2.4

[ ] rabbitmq_mqtt                     3.2.4

[ ] rabbitmq_shovel                   3.2.4

[ ] rabbitmq_shovel_management        3.2.4

[E] rabbitmq_stomp                    3.2.4

[ ] rabbitmq_tracing                  3.2.4

[e] rabbitmq_web_dispatch             3.2.4

[ ] rabbitmq_web_stomp                3.2.4

[ ] rabbitmq_web_stomp_examples       3.2.4

[ ] rfc4627_jsonrpc                   3.2.4-git5e67120

[ ] sockjs                            0.3.4-rmq3.2.4-git3132eb9

[e] webmachine                        1.10.3-rmq3.2.4-gite9359c7

On Tue, Aug 26, 2014 at 5:46 PM, Michael Klishin <[hidden email]> wrote:

 On 27 August 2014 at 04:44:04, srikanth tns ([hidden email]) wrote:
> Is there any optimal value for global / basic policy ?

If you're happy with the current RAM use levels for stats DB, I'd recommend
leaving things as they are for a while.

> Our needs
> are to just monitor the number of connections and use that data
> to load balance our connections.

If you don't use message rates info, you probably don't need fine grained stats.
Again, if things work for you as they are after the change, I'd leave it
this way until the need to re-enable fine grained stats arises.

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

On 27 August 2014 at 04:56:04, srikanth tns ([hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high
> memory , the plugin value is -ve

Do you use STOMP actively?

We've improved some things in STOMP and MQTT plugins in the upcoming
3.4.0 release that would lead to high RAM usage reported by the plugins
(and lower by binaries). This is not a leak but something that skews
the picture quite a bit and results in higher RAM use from "non-core" protocol
connections for no good reason.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Michael Klishin-4]

 On 27 August 2014 at 04:56:04, srikanth tns ([hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high  
> memory , the plugin value is -ve

And the negative value is indeed a bug, 26058, which was fixed in 3.3.0:
http://www.rabbitmq.com/release-notes/README-3.3.0.txt
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[srikanth tns]

Yes we use STOMP actively to send message to the targets hosts via mcollective rpc. The messages are sent to the targets every 10 secs.

Does it cause any issue ?

On Tue, Aug 26, 2014 at 6:01 PM, Michael Klishin <[hidden email]> wrote:

 On 27 August 2014 at 04:56:04, srikanth tns ([hidden email]) wrote:
> > Great one of the things I saw when the rabbitmq was running high
> memory , the plugin value is -ve

And the negative value is indeed a bug, 26058, which was fixed in 3.3.0:
http://www.rabbitmq.com/release-notes/README-3.3.0.txt

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss