ulimit and cgroup

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ulimit and cgroup

ratheesh kannoth
Suppose i restrict  REZ memory and file size   etc ( system resources
)  using  ulimit and cgroup, what will be the behaviour of rabbitMQ. ?
 Suppose a lot of messages gets queued and writing to disk ; if i have
a limit on the file size. ??

-Ratheesh
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

Michael Klishin-2
On 13 Mar 2014, at 07:58, ratheesh kannoth <[hidden email]> wrote:

> Suppose i restrict  REZ memory and file size   etc ( system resources
> )  using  ulimit and cgroup, what will be the behaviour of rabbitMQ. ?
> Suppose a lot of messages gets queued and writing to disk ; if i have
> a limit on the file size. ??

It depends on message size but once you cross a certain threshold, RabbitMQ
will try to move messages to disk. There should be no giant files, though, on disk
message store will split them into multiple smaller files (16 MB by default, AFAIR).

There is a fixed (a few dozens of bytes) RAM cost per message until you use a
custom queue index plugin, such as rabbitmq-toke [1].

If RabbitMQ cannot allocate memory or write to a file, of course, there is little it can do, so be
careful about setting RSS and file size limit. It’s a good idea to adjust RabbitMQ VM high memory watermark
to be lower than the OS limit you use.

1. http://hg.rabbitmq.com/rabbitmq-toke/

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

ratheesh kannoth
On Thu, Mar 13, 2014 at 2:03 PM, Michael Klishin <[hidden email]> wrote:
>>
> It depends on message size but once you cross a certain threshold, RabbitMQ
> will try to move messages to disk.
This is the case even if  we don't use durable queues...right ?

> There should be no giant files, though, on disk
> message store will split them into multiple smaller files (16 MB by default, AFAIR).

All these files would be opened and kept. Or will be opened on demand.
Because linux ulimit can set limit on
no of opened files.

> There is a fixed (a few dozens of bytes) RAM cost per message until you use a
> custom queue index plugin, such as rabbitmq-toke [1].

Can you list out other plugins ( and tuning parameters ) through which
i can tune Broker's system resource usage ?

> If RabbitMQ cannot allocate memory or write to a file, of course, there is little it can do, so be
> careful about setting RSS and file size limit.
Will it gracefully shutdown ?  Can you predict the behavior ? it just crash ?

-Ratheesh
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

Michael Klishin-2
On 13 Mar 2014, at 12:47, ratheesh kannoth <[hidden email]> wrote:

> This is the case even if  we don't use durable queues...right ?

This is true for non-persistent messages, too.

>
>> There should be no giant files, though, on disk
>> message store will split them into multiple smaller files (16 MB by default, AFAIR).
>
> All these files would be opened and kept. Or will be opened on demand.
> Because linux ulimit can set limit on
> no of opened files.

They are created as needed. However, during compaction (garbage collection
of messages on disk), the number of files may temporarily become up to 2 times greater
than usual.

It is rarely a good idea to set a limit on how many files RabbitMQ (or any other infrastructure service, e.g. PostgreSQL or Cassandra)
can open. Best case scenario it will stop accepting new connections (there will be messages
in the log about this).

In general RabbitMQ tries to reduce the number of file descriptors it uses.

>
>> There is a fixed (a few dozens of bytes) RAM cost per message until you use a
>> custom queue index plugin, such as rabbitmq-toke [1].
>
> Can you list out other plugins ( and tuning parameters ) through which
> i can tune Broker's system resource usage ?

http://www.rabbitmq.com/configure.html#configuration-file

>
>> If RabbitMQ cannot allocate memory or write to a file, of course, there is little it can do, so be
>> careful about setting RSS and file size limit.
> Will it gracefully shutdown ?  Can you predict the behavior ? it just crash ?

If it fails to write to a file, the OS process won’t crash. If VM fails to allocate memory,
it will exit and RabbitMQ will have no chance to shut down.

Graceful shutdown involves disk I/O, too.
--
MK

Software Engineer, Pivotal/RabbitMQ

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

Matthias Radestock-3
On 13/03/14 09:18, Michael Klishin wrote:
> On 13 Mar 2014, at 12:47, ratheesh kannoth <[hidden email]> wrote:
>> All these files would be opened and kept. Or will be opened on demand.
>> Because linux ulimit can set limit on
>> no of opened files.
>
> In general RabbitMQ tries to reduce the number of file descriptors it uses.

...and it *does* stick to the ulimit; it is possible to run rabbit with
just a handful of file descriptors, but this will severely impact
performance.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

ratheesh kannoth
On Thu, Mar 13, 2014 at 2:55 PM, Matthias Radestock
<[hidden email]> wrote:
> ...and it *does* stick to the ulimit; it is possible to run rabbit with just
> a handful of file descriptors, but this will severely impact performance.
Is there a configuration to choose  no of descriptor ? How is this handled ?

On Thu, Mar 13, 2014 at 2:03 PM, Michael Klishin <[hidden email]> wrote
>http://www.rabbitmq.com/configure.html#configuration-file
I saw a paramter  "cluster_nodes" . it mentions about automatic
clustering. So this means, clustering just works between brokers.And
producer and consumer are unaware of chosen broker  and  clustering of
brokers ?. I am just curious here. Because ultimately these
connections are over tcp, and are  initiated from client to server.
Atleast ; there should be an communication  from server to client
saying that :-" hey i am down , please send it to other node" . Is
part is already implemented  by rabbitmq clients ?

 -Ratheesh
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

Matthias Radestock-3
On 13/03/14 09:36, ratheesh kannoth wrote:
> On Thu, Mar 13, 2014 at 2:55 PM, Matthias Radestock
> <[hidden email]> wrote:
>> ...and it *does* stick to the ulimit; it is possible to run rabbit with just
>> a handful of file descriptors, but this will severely impact performance.
> Is there a configuration to choose  no of descriptor ? How is this handled ?

Rabbit is aware of the ulimit. So to adjust the max number of file
descriptors rabbit will use, simply adjust the ulimit.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

ratheesh kannoth
On Thu, Mar 13, 2014 at 3:11 PM, Matthias Radestock
<[hidden email]> wrote:

> On 13/03/14 09:36, ratheesh kannoth wrote:
>>
>> On Thu, Mar 13, 2014 at 2:55 PM, Matthias Radestock
>> <[hidden email]> wrote:
>>>
>>> ...and it *does* stick to the ulimit; it is possible to run rabbit with
>>> just
>>> a handful of file descriptors, but this will severely impact performance.
>>
>> Is there a configuration to choose  no of descriptor ? How is this handled
>> ?
>
>
> Rabbit is aware of the ulimit. So to adjust the max number of file
> descriptors rabbit will use, simply adjust the ulimit.
>
>
> Matthias.


Thanks.  Any comment on my  question on clustering ( in my previous email ) ?


-Ratheesh
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ulimit and cgroup

michaelklishin
In reply to this post by ratheesh kannoth

2014-03-13 13:36 GMT+04:00 ratheesh kannoth <[hidden email]>:
And
producer and consumer are unaware of chosen broker  and  clustering of
brokers ?. I am just curious here. Because ultimately these
connections are over tcp, and are  initiated from client to server.
Atleast ; there should be an communication  from server to client
saying that :-" hey i am down , please send it to other node" . Is
part is already implemented  by rabbitmq clients ?

It's a good idea to not mix two completely unrelated questions in the same thread. 

Java client in 3.3 will have automatic recovery + it can use a list of addresses.

Ruby clients and Langohr [1] have automatic recovery but do support connection
to a list of nodes. Most of other clients do no support automatic connection recovery.

Finally, none of the clients internally buffer messages (in RAM or on disk) for redelivery in case
connection goes down. Projects that build on top of clients may do this (e.g. Hutch, federation
plugin).
--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss