Too Many Open Files - Already set max files

Elastic Stack Elasticsearch
1

I have followed the guidelines for setting maximum open file descriptors to 65k and verified that they are set that way on my cluster, however after a crash last night where we lost a master node, our cluster will not recover and is throwing tons of "Too many open files" errors.

Example error:

[2016-11-01 15:24:38,779][WARN ][cluster.action.shard ] [NODE] [INDEX1][0] received shard failed for [INDEX1][0], node[7ASpd2rMT1iORDwmSE_7Ug], [R], v[82], s[STARTED], a[id=G2r6AvbyQkiE9T9Qo5kAtA], indexUUID [T9woubwETvGmWAuLQn9xpA], message [failed to perform indices:data/write/bulk[s] on replica on node {NODE}{7ASpd2rMT1iORDwmSE_7Ug}{IP}{IP:9300}{master=false}], failure [RemoteTransportException[[NODE][IP:9300][indices:data/write/bulk[s][r]]]; nested: ElasticsearchException[failed to sync translog]; nested: NotSerializableExceptionWrapper[PATH/translog/translog.ckp: Too many open files]; ]

Example node stats:

"7ASpd2rMT1iORDwmSE_7Ug" : {
  "timestamp" : 1478013569336,
  "name" : "NODE_NAME",
  "transport_address" : "IP:9300",
  "host" : "NODE_NAME",
  "ip" : [ "IP:9300", "NONE" ],
  "attributes" : {
    "master" : "false"
  },
  "process" : {
    "timestamp" : 1478013569336,
    "open_file_descriptors" : 64858,
    "max_file_descriptors" : 65000,
    "cpu" : {
      "percent" : 97,
      "total_in_millis" : 1470798750
    },
    "mem" : {
      "total_virtual_in_bytes" : 27855085568
    }
  }
}

What kind of change to my cluster does this indicate I need to make for stability?? Do we need more nodes?? Do we need less indices & shards??

Is there an api call to up the maximum number of file descriptors on a live elasticsearch process?? Or is the only way to change the system settings and then restart elasticsearch? I'd like to avoid having to restart data nodes if possible.

Help appreciated... Thank you!

2

Can you give some more information about your current setup, e.g. how many nodes, indices, shards per Index do you have? What's your primary use case? What amount of data do you usually ingest, how many reads do you get on average?

Regarding your question: I think the only way to update this setting is indeed to restart the nodes one after the other.

3

Topology:
3 Master nodes
2 Client nodes (for data intake)
2 Client nodes (for querying)
4 Data nodes (now 8, we added 4 after the crash)

All machines 4cpu's and 15gb memory.

1612 indices with 10 shards each and 1 replica
Total data size 662GB
Daily Intake ~ 40gb

Elastalert runs about 30 varying queries every 3 minutes against dedicated client nodes.
Devs run queries and aggregations against the cluster regularly via kibana.

I guess I'll follow the rolling restarts instructions for the maximum file limit setting.

4

You are wasting a lot of resources by having far too many shards and indices. For use cases with time based data we generally see average shard sizes of at least a few GB in size. If I have calculated correctly, your average shard size is around 20MB which is very, very small.

I would recommend dramatically reducing the number of shards for each index and possibly also look at consolidate indices. This will allow you get a lot more from your cluster.

1 Like
5

That's what I was starting to suspect. We're thinking of a new way to consolidate our indices then.

Thank you!!