Elasticsearch: Too many open files error even with 64K max file descriptor set

Elastic Stack Elasticsearch
1

I have a 3 node cluster (2 data/master and 1 master/no data. It is being fed by 2 logstash servers. I have everything tuned based on the production setting suggestions (max file descriptors set to 64K, setting ES Heap Size, turning off swap, enabling mlockall). Everything was running fine until it just seemingly locked up and I started getting "Too many open files" messages in the elasticsearch.log.

Is there a way to tell what pushed the data servers over the edge? Are there other performance settings to look at?

2

Maybe you have an insanely high number of shards on your machine?

3

These are the current statistics:

status" : "green",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 2,
"active_primary_shards" : 281,
"active_shards" : 562,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"number_of_pending_tasks" : 0

Is that too many shards?

4

This is a bit high but not insane. I'm very surprised that you manage to reach the max number of file descriptors with that many shards. Any chance that you can count how many files you have in your data directories? Also can you tell us the maximum number of file descriptors as seen by elasticsearch? https://www.elastic.co/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html

5

"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 1418,
"max_file_descriptors" : 64000,
"mlockall" : true

find '/usr/local/elasticsearch/data/' -type f | wc -l
12400

Which is below the 64000 limit.