Changing file descriptor limit wihtout restart?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Changing file descriptor limit wihtout restart?

Jakov Sosic
Hi guys


I want to increase file descriptor limit (ulimit -n) on my running
RabbitMQ cluster.

I know I can do it by setting up limits.conf and restarting RabbitMQ
nodes but I would prefer to do it without node downtime if possible.

What I tried was:

# echo -n "Max open files=4096:4096" > /proc/$(cat
/var/run/rabbitmq/pid)/limits

but rabbitmqctl still shows lower value:

# rabbitmqctl status | grep -A 2 file_descriptors
  {file_descriptors,
      [{total_limit,924},
       {total_used,163},


Where is this total limit of 924 coming from at first place?!

Process itself seems to accepted the increase :

# grep -i open /proc/$(cat /var/run/rabbitmq/pid)/limits
Max open files          4096               4096               files
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Michael Klishin-2
On 5 July 2014 at 22:56:11, Jakov Sosic ([hidden email]) wrote:
> > I know I can do it by setting up limits.conf and restarting RabbitMQ  
> nodes but I would prefer to do it without node downtime if possible.  

ulimit changes will only be effective after OS process restart.
This is true for all processes, RabbitMQ or not. 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Jakov Sosic
On 07/05/2014 08:58 PM, Michael Klishin wrote:
> On 5 July 2014 at 22:56:11, Jakov Sosic ([hidden email]) wrote:
>>> I know I can do it by setting up limits.conf and restarting RabbitMQ
>> nodes but I would prefer to do it without node downtime if possible.
>
> ulimit changes will only be effective after OS process restart.
> This is true for all processes, RabbitMQ or not.

What about write-able limits and prlimit(2)?

Although it seems writeable limits is not affecting running programs, at
least not on CentOS 6... I have written small C program to verify this:

#include <stdio.h>
#include <limits.h>
#include <unistd.h>

int main(){
        int i=sysconf(_SC_OPEN_MAX);
        int n=0;

        while(n<1000){
                n++;
                printf("Max open files: %d\n", i);
                sleep(5);
        }
        return 0;
}


but, after running it and increasing ulimit via

echo -n "Max open files=4096:4096" > /proc/<pid>/limits

program still reports 1024 :-/
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Tony Garnock-Jones-6
Try putting the call to sysconf() inside the loop.


On 5 July 2014 16:26, Jakov Sosic <[hidden email]> wrote:
On 07/05/2014 08:58 PM, Michael Klishin wrote:
On 5 July 2014 at 22:56:11, Jakov Sosic ([hidden email]) wrote:
I know I can do it by setting up limits.conf and restarting RabbitMQ
nodes but I would prefer to do it without node downtime if possible.

ulimit changes will only be effective after OS process restart.
This is true for all processes, RabbitMQ or not.

What about write-able limits and prlimit(2)?

Although it seems writeable limits is not affecting running programs, at least not on CentOS 6... I have written small C program to verify this:

#include <stdio.h>
#include <limits.h>
#include <unistd.h>

int main(){
        int i=sysconf(_SC_OPEN_MAX);
        int n=0;

        while(n<1000){
                n++;
                printf("Max open files: %d\n", i);
                sleep(5);
        }
        return 0;
}


but, after running it and increasing ulimit via

echo -n "Max open files=4096:4096" > /proc/<pid>/limits

program still reports 1024 :-/
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
Tony Garnock-Jones
[hidden email]
http://homepages.kcbbs.gen.nz/tonyg/

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Jakov Sosic
On 07/05/2014 11:40 PM, Tony Garnock-Jones wrote:
> Try putting the call to sysconf() inside the loop.

That's what happens when you don't get much sleep :)


Anyway it works:

Max open files: 1024
Max open files: 2048
Max open files: 2048
Max open files: 4096


which brings us back to original question, is there a way to increase
the descriptor limit for running RabbitMQ server? Is there a way for
ERLang process to figure out limits have changed?
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Simon MacMullen-2
On 06/07/2014 12:03AM, Jakov Sosic wrote:
> which brings us back to original question, is there a way to increase
> the descriptor limit for running RabbitMQ server? Is there a way for
> ERLang process to figure out limits have changed?

It's not Erlang, it's RabbitMQ. The problem is that the file handle
cache (the component which manages file descriptors inside the broker)
only checks the limit at startup.

But you can tell it the limit has changed with something like:

$ rabbitmqctl eval 'file_handle_cache:set_limit(4096).'

BIG FAT WARNING:

This number is not validated; it's taken entirely on trust. If you tell
the FHC that it has a larger limit than the one the OS will enforce, you
can run out of file descriptors and crash the server.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Jakov Sosic
On 07/07/2014 11:35 AM, Simon MacMullen wrote:

> It's not Erlang, it's RabbitMQ. The problem is that the file handle
> cache (the component which manages file descriptors inside the broker)
> only checks the limit at startup.
>
> But you can tell it the limit has changed with something like:
>
> $ rabbitmqctl eval 'file_handle_cache:set_limit(4096).'
>
> BIG FAT WARNING:
>
> This number is not validated; it's taken entirely on trust. If you tell
> the FHC that it has a larger limit than the one the OS will enforce, you
> can run out of file descriptors and crash the server.



That will increase "Network sockets count and limit managed by
RabbitMQ". File descriptor count stays hard locked at 1024.

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Changing file descriptor limit wihtout restart?

Simon MacMullen-2
On 09/09/14 20:56, Jakov Sosic wrote:
> That will increase "Network sockets count and limit managed by
> RabbitMQ". File descriptor count stays hard locked at 1024.

Ah. It looks like the old value is cached in the management plugin
indefinitely.

However, if you look at "rabbitmqctl status" then you will see the
correct new value. And that's the number the file handle cache will
actually take account of.

Cheers, Simon
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Loading...