Too many open files: does a "service wrapper" end that issue?

jackpark · December 30, 2013, 2:01am

The tutorial at http://www.elasticsearch.org/tutorials/too-many-open-files/
says this:

If sudo -u elasticsearch -s "ulimit -Sn" shows 32000 but you
still have a low limit when you run ElasticSearch, you’re probably running it
through another program that doesn’t support PAM: a frequent offender is
supervisord.

[I have no way to know I'm running through supervisord, since I'm just typing the ./elasticsearch -f into a terminal on Ubuntu]

The only solution I know to this problem is to raise the nofile
limit for the user running supervisord, but this will obviously raise the limit
for all the processes running under supervisord, not an ideal situation.

Consider using the
ElasticSearch
service wrapper instead.

This is the service wrapper mentioned:

Is a service wrapper going to put this issue to bed?

Is there a way to ask ElasticSearch how many open files Lucene is using?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyjTm%2BYN69PhgPSwObyJo6bbZa1Ac%2BzLK2J4i5oxyO5Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 30, 2013, 2:55am

Have a look at: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html

About service wrapper, I would recommend to use RPM, DEB packages.

You can display current Max open files setting when starting elasticsearch using https://github.com/elasticsearch/elasticsearch/issues/483

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 déc. 2013 à 03:01, Jack Park jackpark@topicquests.org a écrit :

The tutorial at http://www.elasticsearch.org/tutorials/too-many-open-files/
says this:

If sudo -u elasticsearch -s "ulimit -Sn" shows 32000 but you
still have a low limit when you run ElasticSearch, you’re probably running it
through another program that doesn’t support PAM: a frequent offender is
supervisord.

[I have no way to know I'm running through supervisord, since I'm just typing the ./elasticsearch -f into a terminal on Ubuntu]

The only solution I know to this problem is to raise the nofile
limit for the user running supervisord, but this will obviously raise the limit
for all the processes running under supervisord, not an ideal situation.

Consider using the
ElasticSearch
service wrapper instead.

This is the service wrapper mentioned:

Is a service wrapper going to put this issue to bed?

Is there a way to ask ElasticSearch how many open files Lucene is using?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyjTm%2BYN69PhgPSwObyJo6bbZa1Ac%2BzLK2J4i5oxyO5Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E15BDF75-6FA1-45BA-A75C-6F35A7956268%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

jackpark · December 30, 2013, 3:07am

Yes!
-Des.max-open-files=true does show me it can take 32000 files.

But,
typing this in

http://localhost:9200/_nodes/stats
on that machine says "No such file or directory"

That would be cool if there's an easy way to see what the stats are,
and perhaps stop and optimize or something.

Many thanks!

On Sun, Dec 29, 2013 at 6:55 PM, David Pilato david@pilato.fr wrote:

Have a look at:
Elasticsearch Platform — Find real-time answers at scale | Elastic

About service wrapper, I would recommend to use RPM, DEB packages.

You can display current Max open files setting when starting elasticsearch
using Add an option to display max_open_files, by setting -Des.max-open-files to `true` · Issue #483 · elastic/elasticsearch · GitHub

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 déc. 2013 à 03:01, Jack Park jackpark@topicquests.org a écrit :

The tutorial at Elasticsearch Platform — Find real-time answers at scale | Elastic
says this:

If sudo -u elasticsearch -s "ulimit -Sn" shows 32000 but you
still have a low limit when you run Elasticsearch, you’re probably running
it
through another program that doesn’t support PAM: a frequent offender is
supervisord.

[I have no way to know I'm running through supervisord, since I'm just typing the ./elasticsearch -f into a terminal on Ubuntu]

The only solution I know to this problem is to raise the nofile
limit for the user running supervisord, but this will obviously raise the
limit
for all the processes running under supervisord, not an ideal situation.

Consider using the
Elasticsearch
service wrapper instead.

This is the service wrapper mentioned:
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

Is a service wrapper going to put this issue to bed?

Is there a way to ask Elasticsearch how many open files Lucene is using?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyjTm%2BYN69PhgPSwObyJo6bbZa1Ac%2BzLK2J4i5oxyO5Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E15BDF75-6FA1-45BA-A75C-6F35A7956268%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwupvSXGXx%2BRj5HsQ5aEqteNO50tRaEmAnfFtf3C8O2Ag%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jackpark · December 30, 2013, 4:12am

Something I did not notice earlier; in one of the trials, the system
came up saying it could open 32000 files. Now it is saying 9400.
ulimit -n says 32000, and the system was rebooted. Why would ES not be
picking up that limit?

Thanks.
Jack

On Sun, Dec 29, 2013 at 7:07 PM, Jack Park jackpark@topicquests.org wrote:

Yes!
-Des.max-open-files=true does show me it can take 32000 files.

But,
typing this in

http://localhost:9200/_nodes/stats
on that machine says "No such file or directory"

That would be cool if there's an easy way to see what the stats are,
and perhaps stop and optimize or something.

Many thanks!

On Sun, Dec 29, 2013 at 6:55 PM, David Pilato david@pilato.fr wrote:

Have a look at:
Elasticsearch Platform — Find real-time answers at scale | Elastic

About service wrapper, I would recommend to use RPM, DEB packages.

You can display current Max open files setting when starting elasticsearch
using Add an option to display max_open_files, by setting -Des.max-open-files to `true` · Issue #483 · elastic/elasticsearch · GitHub

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 déc. 2013 à 03:01, Jack Park jackpark@topicquests.org a écrit :

The tutorial at Elasticsearch Platform — Find real-time answers at scale | Elastic
says this:

If sudo -u elasticsearch -s "ulimit -Sn" shows 32000 but you
still have a low limit when you run Elasticsearch, you’re probably running
it
through another program that doesn’t support PAM: a frequent offender is
supervisord.

[I have no way to know I'm running through supervisord, since I'm just typing the ./elasticsearch -f into a terminal on Ubuntu]

The only solution I know to this problem is to raise the nofile
limit for the user running supervisord, but this will obviously raise the
limit
for all the processes running under supervisord, not an ideal situation.

Consider using the
Elasticsearch
service wrapper instead.

This is the service wrapper mentioned:
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

Is a service wrapper going to put this issue to bed?

Is there a way to ask Elasticsearch how many open files Lucene is using?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyjTm%2BYN69PhgPSwObyJo6bbZa1Ac%2BzLK2J4i5oxyO5Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E15BDF75-6FA1-45BA-A75C-6F35A7956268%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy6798KH6ps8jQZsg-j9St%3D3SPSzZT%2BAp9v%2BchDAp0L1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · December 30, 2013, 7:53am

Are you running ES on a virtual machine in a guest OS?

The service wrapper
https://github.com/elasticsearch/elasticsearch-servicewrapper works well
but it is not related to file descriptor limit setting. This is a case for
the host OS.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGfJ8phUPakj6fevoU4pGbYWpHQWamwAuL%3DX4k0XajBTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jackpark · December 30, 2013, 3:10pm

Running bare on various Ubuntu boxes.
Interesting finding still deep in testing;
For Ubuntu, what does work is this:
Add this line to elasticsearch.in.sh:
ulimit -n 32000 (or whatever)
The notion of doing what the es tutorial says seems required, but
insufficient when on Ubuntu: even after doing all of that, and
rebooting, es with -Des.max-open-files=true is still not "getting it".
One obscure comment somewhere on the web suggested adding that line in
the shell, more as an offhand comment -- no theory, no reason, and now
es boots with the ability to open near that number (31999 or
something). So, it is now running on two different platforms to see if
my import will survive. One failed overnight with the es console
saying "Killed" for no apparent reason. Just "Killed", which, of
course, hosed that import. The other one is still running.

On Sun, Dec 29, 2013 at 11:53 PM, joergprante@gmail.com
joergprante@gmail.com wrote:

Are you running ES on a virtual machine in a guest OS?

The service wrapper
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch works well but
it is not related to file descriptor limit setting. This is a case for the
host OS.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGfJ8phUPakj6fevoU4pGbYWpHQWamwAuL%3DX4k0XajBTQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fz_8znxUhNG3KCvA2EQxZdAz9wvP0ukfuaWHRY6Paj7Fg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · December 30, 2013, 3:27pm

Have you checked /var/log/messages for a "killed" message?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoELH0mcK6xDXah2jhWGo8F5sE3nDSrYLgrLXEiCZOkLtg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jackpark · December 30, 2013, 3:50pm

These lines in that log
Dec 30 03:37:11 bloomer kernel: [24733.355784] []
oom_kill_process+0x81/0x180
Dec 30 03:37:11 bloomer kernel: [24733.355786] []
__out_of_memory+0x58/0xd0
Dec 30 03:37:11 bloomer kernel: [24733.355788] []
out_of_memory+0x86/0x1c0

suggest something about out of memory.
On that box, I was only able to use 4g. The log suggests perhaps
that's not enough.

On Mon, Dec 30, 2013 at 7:27 AM, joergprante@gmail.com
joergprante@gmail.com wrote:

Have you checked /var/log/messages for a "killed" message?

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoELH0mcK6xDXah2jhWGo8F5sE3nDSrYLgrLXEiCZOkLtg%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxb%2Br2_4c054oecNcej2HxPpai0kZPU%2B5KHEp3Qis5eeA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · December 30, 2013, 3:57pm

The log says, the kernel OOM killer has stopped your process because of
getting too big. You have to configure less memory for the JVM or put in
more RAM.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGkXwXOJ%2BedX%2BQXubseQyT4dwsbKV%3DptRuq7TuV6zeNYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.