Memory not flushing?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory not flushing?

mc717990
Running 3.3.4, we have a situation (maybe an erlang R14B04 issue?) where rabbit hits the memory alarm and never goes out of it.  The queue (with over 100 million messages in it) is showing 0 memory usage, all messages paged to disk, 0 in ram.  Disk IO is non existent.  Just looking for any ideas/suggestions?  I saw this message:
But wasn't sure it applied since the ram counts are 0?  If that's still probably the issue, is there a way to set the target ram count to something other than infinity?  e.g. rabbitmqctl set_paramter myqueue target_ram_count 0.50?  Below are the memory reports from a few place.  
Thanks!
Jason

#####
Memory report on the node via the api memory=true parameter:

  • {
    • total14309117272,
    • connection_procs1406816,
    • queue_procs10781072,
    • plugins637448,
    • other_proc9246264,
    • mnesia66312,
    • mgmt_db316096,
    • msg_index12411588360,
    • other_ets1844522120,
    • binary9166464,
    • code17795943,
    • atom1619705,
    • other_system1970672
    },
  • partitions: [ ],
  • os_pid"23782",
  • fd_used53,
  • fd_total1024,
  • sockets_used27,
  • sockets_total829,
  • mem_used14309008432,
  • mem_limit1682936627,
  • mem_alarmtrue,
  • disk_free_limit50000000,
  • disk_free362680954880,
  • disk_free_alarmfalse,
  • proc_used472,
  • proc_total1048576,
  • statistics_level"fine",
  • uptime88779083,
  • run_queue0,
  • processors4,
  • exchange_types
    [
####
Backing queue status from the queue (only one queue):
  • backing_queue_status
    {
    • q10,
    • q20,
    • delta
      [
      • "delta",
      • 25034752,
      • 101610341,
      • 126670623
      ],
    • q33667,
    • q40,
    • len101614008,
    • pending_acks0,
    • target_ram_count"infinity",
    • ram_msg_count0,
    • ram_ack_count0,
    • next_seq_id126670623,
    • persistent_count101614008,
    • avg_ingress_rate6.316340263548813e-276,
    • avg_egress_rate0,
    • avg_ack_ingress_rate0,
    • avg_ack_egress_rate0
    },

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Memory not flushing?

Michael Klishin-2
On 9 July 2014 at 23:25:12, Jason McIntosh ([hidden email]) wrote:
> > Running 3.3.4, we have a situation (maybe an erlang R14B04 issue?)  
> where rabbit hits the memory alarm and never goes out of it. The  
> queue (with over 100 million messages in it) is showing 0 memory  
> usage, all messages paged to disk, 0 in ram. Disk IO is non existent.  
> Just looking for any ideas/suggestions?

Unless you use the Tokyo Cabinet-based message index implementation [1], there's a fixed
RAM cost per message (something between 15-20 bytes AFAIR). 

1. http://hg.rabbitmq.com/rabbitmq-toke/
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Memory not flushing?

Michael Klishin-2
In reply to this post by mc717990
On 9 July 2014 at 23:25:12, Jason McIntosh ([hidden email]) wrote:
> > msg_index: 12411588360,

That's about 11.5  GB used by the message index.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Memory not flushing?

Michael Klishin-2
In reply to this post by mc717990
 On 9 July 2014 at 23:29:54, Michael Klishin ([hidden email]) wrote:
> > there's a fixed
> RAM cost per message (something between 15-20 bytes AFAIR)

Nope, it's more like ~ 100 bytes. 11.5 GB then sounds about right for ~ 100M messages.

We need to document this :)
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Memory not flushing?

Simon MacMullen-2
In reply to this post by mc717990
I think Michael already answered your real question - you are running
into the msg store index using all memory. But to answer some of your
sub-questions...

On 09/07/2014 8:22PM, Jason McIntosh wrote:
> I saw this message:
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-April/026649.html
> But wasn't sure it applied since the ram counts are 0?

No, it doesn't. If the ram count is 0, all messages are paged out.

> If that's still
> probably the issue, is there a way to set the target ram count to
> something other than infinity?  e.g. rabbitmqctl set_paramter myqueue
> target_ram_count 0.50?

The target ram count is managed automatically. Ultimately it's derived
from vm_memory_high_watermark_paging_ratio, via a system which attempts
to ensure that all queues get a fair share of memory, where "fair" is
defined as "slow-moving queues page out earlier than fast-moving queues,
since they're slow anyway". It's a feedback-based system though, which
means that if all queues are not accepting messages (i.e. when the
memory alarm goes off) then it can end up saying "infinity". If the
queue started moving again that number would go down some (and it's
clearly been 0 in the past, hence the ram_msg_count).

Cheers, Simon

> Below are the memory reports from a few place.
> Thanks!
> Jason
>
> #####
> Memory report on the node via the api memory=true parameter:
>
>   *
>     {
>       o
>         total: 14309117272,
>       o
>         connection_procs: 1406816,
>       o
>         queue_procs: 10781072,
>       o
>         plugins: 637448,
>       o
>         other_proc: 9246264,
>       o
>         mnesia: 66312,
>       o
>         mgmt_db: 316096,
>       o
>         msg_index: 12411588360,
>       o
>         other_ets: 1844522120,
>       o
>         binary: 9166464,
>       o
>         code: 17795943,
>       o
>         atom: 1619705,
>       o
>         other_system: 1970672
>     },
>   *
>     partitions: [ ],
>   *
>     os_pid: "23782",
>   *
>     fd_used: 53,
>   *
>     fd_total: 1024,
>   *
>     sockets_used: 27,
>   *
>     sockets_total: 829,
>   *
>     mem_used: 14309008432,
>   *
>     mem_limit: 1682936627,
>   *
>     mem_alarm: true,
>   *
>     disk_free_limit: 50000000,
>   *
>     disk_free: 362680954880,
>   *
>     disk_free_alarm: false,
>   *
>     proc_used: 472,
>   *
>     proc_total: 1048576,
>   *
>     statistics_level: "fine",
>   *
>     uptime: 88779083,
>   *
>     run_queue: 0,
>   *
>     processors: 4,
>   *
>     exchange_types:
>     [
>
> ####
> Backing queue status from the queue (only one queue):
>
>   *
>     backing_queue_status:
>     {
>       o
>         q1: 0,
>       o
>         q2: 0,
>       o
>         delta:
>         [
>           +
>             "delta",
>           +
>             25034752,
>           +
>             101610341,
>           +
>             126670623
>         ],
>       o
>         q3: 3667,
>       o
>         q4: 0,
>       o
>         len: 101614008,
>       o
>         pending_acks: 0,
>       o
>         target_ram_count: "infinity",
>       o
>         ram_msg_count: 0,
>       o
>         ram_ack_count: 0,
>       o
>         next_seq_id: 126670623,
>       o
>         persistent_count: 101614008,
>       o
>         avg_ingress_rate: 6.316340263548813e-276,
>       o
>         avg_egress_rate: 0,
>       o
>         avg_ack_ingress_rate: 0,
>       o
>         avg_ack_egress_rate: 0
>     },
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
> please subscribe to the new list!
>
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss