Has the number of files open at once by ES increased dramatically in
v0.11 or v0.12? Code that previously ran under v0.10 is now
consistently crashing with "too many open files" errors, and cat /proc/
sys/fs/file-max on the system ES is running on is reporting over 1M
file descriptors available.
ulimit is a better number to get as file-max is for the entire system
whereas ulimit is per process.
To see how many files elasticsearch has open, simply do an lsof.
lsof -p <process_id_of_es> | wc -l
How many does this come up with?
On Tue, Oct 19, 2010 at 11:13 PM, Adam Parkin pzelnip@gmail.com wrote:
Has the number of files open at once by ES increased dramatically in
v0.11 or v0.12? Code that previously ran under v0.10 is now
consistently crashing with "too many open files" errors, and cat /proc/
sys/fs/file-max on the system ES is running on is reporting over 1M
file descriptors available.
--
Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy
Hi,
yes, the format the index is stored was changed from compound format to
non compound format in v0.11. Here is the issue:
Issues · elastic/elasticsearch · GitHub (it
also explains how to set back to using the non compound format). The reason
for this is better performance and IO behavior when using the non compound
format, at the expense of more file descriptors open.
Can you check with ulimit? How many indices and shards are you creating?
How many machines do you have? What is your ulimit setting now? Bumping it
up to 32k is usually good for most systems.
-shay.banon
On Wed, Oct 20, 2010 at 12:28 AM, Paul Loy keteracel@gmail.com wrote:
ulimit is a better number to get as file-max is for the entire system
whereas ulimit is per process.To see how many files elasticsearch has open, simply do an lsof.
lsof -p <process_id_of_es> | wc -l
How many does this come up with?
On Tue, Oct 19, 2010 at 11:13 PM, Adam Parkin pzelnip@gmail.com wrote:
Has the number of files open at once by ES increased dramatically in
v0.11 or v0.12? Code that previously ran under v0.10 is now
consistently crashing with "too many open files" errors, and cat /proc/
sys/fs/file-max on the system ES is running on is reporting over 1M
file descriptors available.--
Paul Loy
paul@keteracel.com
Paul Loy - Amihan Entertainment | LinkedIn