Elastic search not detecting file descriptors limit


(William Jimenez) #1

Hi
I have set my file descriptors limit to 400000 on my Ubuntu 10.04 LTS
machine, but elastic search doesn't seem to detect it.

wjimenez@[myhost]> sysctl -a |grep fs.file-max
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'fs.binfmt_misc.register'
error: permission denied on key 'dev.parport.parport0.autoprobe'
error: permission denied on key 'dev.parport.parport0.autoprobe0'
error: permission denied on key 'dev.parport.parport0.autoprobe1'
error: permission denied on key 'dev.parport.parport0.autoprobe2'
error: permission denied on key 'dev.parport.parport0.autoprobe3'
fs.file-max = 400000
error: permission denied on key 'net.ipv4.route.flush'

wjimenez@[myhost]> bin/elasticsearch -f -Des.max-open-files=true
[2012-04-03 20:40:28,116][INFO ][bootstrap ] max_open_files
[999]
Anything I am missing here?


(Radu Gheorghe) #2

Hi William,

AFAIK, these limits are per-user. So if you have 400000 for root and
start ES as wjimenez, then the behavior you're describing is normal.

We are setting those limits in the init script, before starting ES.
Another option is to edit /etc/security/limits.conf and set the limits
for the user ES runs with.

And of course, from the console, you can always do ulimit -n 400000
before starting ES manually.

On Apr 4, 6:41 am, William Jimenez wjimenez5...@gmail.com wrote:

Hi
I have set my file descriptors limit to 400000 on my Ubuntu 10.04 LTS
machine, but elastic search doesn't seem to detect it.

wjimenez@[myhost]> sysctl -a |grep fs.file-max
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'fs.binfmt_misc.register'
error: permission denied on key 'dev.parport.parport0.autoprobe'
error: permission denied on key 'dev.parport.parport0.autoprobe0'
error: permission denied on key 'dev.parport.parport0.autoprobe1'
error: permission denied on key 'dev.parport.parport0.autoprobe2'
error: permission denied on key 'dev.parport.parport0.autoprobe3'
fs.file-max = 400000
error: permission denied on key 'net.ipv4.route.flush'

wjimenez@[myhost]> bin/elasticsearch -f -Des.max-open-files=true
[2012-04-03 20:40:28,116][INFO ][bootstrap ] max_open_files
[999]
Anything I am missing here?


(William Jimenez) #3

Hi Radu
Thanks so much for that explanation. so if I use sudo to run
elasticsearch, would I expect to it to pick up that ulmit setting? Also the
ulmit command doesn't work for me (maybe b/c I am on Ubuntu)?

wjimenez@[myhost]> sudo bin/elasticsearch -f -Des.max-open-files=true
[sudo] password for wjimenez:
[2012-04-04 10:55:22,144][INFO ][bootstrap ] max_open_files
[999]
[2012-04-04 10:55:23,870][INFO ][node ] [Bast]
{0.18.7}[5817]: initializing ...
[2012-04-04 10:55:23,891][INFO ][plugins ] [Bast] loaded
[], sites []
wjimenez@[myhost]> sudo ulimit -n 400000
sudo: ulimit: command not found

Thanks

On Wednesday, April 4, 2012 12:37:45 AM UTC-7, Radu Gheorghe wrote:

Hi William,

AFAIK, these limits are per-user. So if you have 400000 for root and
start ES as wjimenez, then the behavior you're describing is normal.

We are setting those limits in the init script, before starting ES.
Another option is to edit /etc/security/limits.conf and set the limits
for the user ES runs with.

And of course, from the console, you can always do ulimit -n 400000
before starting ES manually.

On Apr 4, 6:41 am, William Jimenez wjimenez5...@gmail.com wrote:

Hi
I have set my file descriptors limit to 400000 on my Ubuntu 10.04 LTS
machine, but elastic search doesn't seem to detect it.

wjimenez@[myhost]> sysctl -a |grep fs.file-max
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'fs.binfmt_misc.register'
error: permission denied on key 'dev.parport.parport0.autoprobe'
error: permission denied on key 'dev.parport.parport0.autoprobe0'
error: permission denied on key 'dev.parport.parport0.autoprobe1'
error: permission denied on key 'dev.parport.parport0.autoprobe2'
error: permission denied on key 'dev.parport.parport0.autoprobe3'
fs.file-max = 400000
error: permission denied on key 'net.ipv4.route.flush'

wjimenez@[myhost]> bin/elasticsearch -f -Des.max-open-files=true
[2012-04-03 20:40:28,116][INFO ][bootstrap ]
max_open_files
[999]
Anything I am missing here?


(Radu Gheorghe) #4

Hi William,

If you want to do the thing as root, I think you have to go this path:

$ sudo su

ulimit -n 400000

bin/elasticsearch -f -Des.max-open-files=true

Apparently, you can't set the limit for another user using sudo, even
if you're root:
root# sudo -u radu ulimit -n 400000
sudo: ulimit: command not found

On Apr 4, 8:58 pm, William Jimenez wjimenez5...@gmail.com wrote:

Hi Radu
Thanks so much for that explanation. so if I use sudo to run
elasticsearch, would I expect to it to pick up that ulmit setting? Also the
ulmit command doesn't work for me (maybe b/c I am on Ubuntu)?

wjimenez@[myhost]> sudo bin/elasticsearch -f -Des.max-open-files=true
[sudo] password for wjimenez:
[2012-04-04 10:55:22,144][INFO ][bootstrap ] max_open_files
[999]
[2012-04-04 10:55:23,870][INFO ][node ] [Bast]
{0.18.7}[5817]: initializing ...
[2012-04-04 10:55:23,891][INFO ][plugins ] [Bast] loaded
[], sites []
wjimenez@[myhost]> sudo ulimit -n 400000
sudo: ulimit: command not found

Thanks

On Wednesday, April 4, 2012 12:37:45 AM UTC-7, Radu Gheorghe wrote:

Hi William,

AFAIK, these limits are per-user. So if you have 400000 for root and
start ES as wjimenez, then the behavior you're describing is normal.

We are setting those limits in the init script, before starting ES.
Another option is to edit /etc/security/limits.conf and set the limits
for the user ES runs with.

And of course, from the console, you can always do ulimit -n 400000
before starting ES manually.

On Apr 4, 6:41 am, William Jimenez wjimenez5...@gmail.com wrote:

Hi
I have set my file descriptors limit to 400000 on my Ubuntu 10.04 LTS
machine, but elastic search doesn't seem to detect it.

wjimenez@[myhost]> sysctl -a |grep fs.file-max
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'fs.binfmt_misc.register'
error: permission denied on key 'dev.parport.parport0.autoprobe'
error: permission denied on key 'dev.parport.parport0.autoprobe0'
error: permission denied on key 'dev.parport.parport0.autoprobe1'
error: permission denied on key 'dev.parport.parport0.autoprobe2'
error: permission denied on key 'dev.parport.parport0.autoprobe3'
fs.file-max = 400000
error: permission denied on key 'net.ipv4.route.flush'

wjimenez@[myhost]> bin/elasticsearch -f -Des.max-open-files=true
[2012-04-03 20:40:28,116][INFO ][bootstrap ]
max_open_files
[999]
Anything I am missing here?


(system) #5