Elasticsearch opening way too many (200K) Pipes, FDs

Elastic Stack Elasticsearch
1

We have elasticsearch running in our product, its not a cluster configuration. We have noticed despite maximum FD limit set to 65536, it has opened too many Pipes/

root]# lsof | grep -i elasticsearch | wc -l
204875
root]#

Out of these open files, almost 160000 are pipes and inodes.

java 1966 elasticsearch 92r FIFO 0,9 0t0 34857 pipe
java 1966 elasticsearch 93w FIFO 0,9 0t0 34857 pipe
java 1966 elasticsearch 94u a_inode 0,10 0 1035 [eventpoll]

If we look for particular elasticsearch process under /proc//fd we have about 671 open FD. But I can understand why for elastic user so many other FDs are opened in the system

_nodes/stats/process?filter_path=**.max_file_descriptors
{}{"nodes":{"s8U48FmKSdGal6JyXuNnzg":{"process":{"max_file_descriptors":65535}}}}

Can you please help me understand this behavior and how it can be eliminated?

2

A first guess is the number of shards per node. How many do you have?

3

Hi David,

Its 1

{
"DB1_v2" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"max_result_window" : "1000000",
"creation_date" : "1509099442192",
"analysis" : {
"analyzer" : {
"lower_term" : {
"filter" : "lowercase",
"type" : "custom",
"tokenizer" : "keyword"
}
}
},
"number_of_replicas" : "1",
"uuid" : "jmrPGEK7Twmo6qS6al9A-g",
"version" : {
"created" : "2030499"
}
}
}
},
"DB2_v1" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"max_result_window" : "1000000",
"creation_date" : "1509099442386",
"analysis" : {
"analyzer" : {
"lower_term" : {
"filter" : "lowercase",
"type" : "custom",
"tokenizer" : "keyword"
}
}
},
"number_of_replicas" : "1",
"uuid" : "48X3eMQzTzidE_5Q6KY9bw",
"version" : {
"created" : "2030499"
}
}
}
}
}

4

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

What gives:

GET _cat/shards?v
5

lsof as you have done is not an accurate way to count open file descriptors. For example, as you have done will count memory-mapped portions of files which do not count individually as file descriptors. Additionally, lsof will duplicate the same file descriptor multiple times for multiple threads. The correct way to check the number of file descriptors is the number of entires in /proc/<pid>/fd.

1 Like
7

I agree, but on a idle system, why we should have so may open FDs pointed by elasticsearch. There are several other services but elastic is still contributing about 80% of open FDs in system

8

Sorry for spamming just tried one more time

GET _settings?pretty
{
  "DB1_v2" : {
"settings" : {
  "index" : {
    "number_of_shards" : "1",
    "max_result_window" : "1000000",
    "creation_date" : "1509099442192",
    "analysis" : {
      "analyzer" : {
        "lower_term" : {
          "filter" : "lowercase",
          "type" : "custom",
          "tokenizer" : "keyword"
        }
      }
    },
    "number_of_replicas" : "1",
    "uuid" : "jmrPGEK7Twmo6qS6al9A-g",
    "version" : {
      "created" : "2030499"
    }
  }
}
  },
  "DB2_v1" : {
"settings" : {
  "index" : {
    "number_of_shards" : "1",
    "max_result_window" : "1000000",
    "creation_date" : "1509099442386",
    "analysis" : {
      "analyzer" : {
        "lower_term" : {
          "filter" : "lowercase",
          "type" : "custom",
          "tokenizer" : "keyword"
        }
      }
    },
    "number_of_replicas" : "1",
    "uuid" : "48X3eMQzTzidE_5Q6KY9bw",
    "version" : {
      "created" : "2030499"
    }
  }
}
  }
}
9 
_cat/shards?v
index   shard prirep   state           docs     store        ip        node
DB2_v1   0     p          STARTED          0       159b 127.0.0.1 Cletus Kasady
DB2_v1   0     r        UNASSIGNED
DB1_v2   0     p         STARTED      4686     2.1mb 127.0.0.1 Cletus Kasady
DB1_v2   0     r        UNASSIGNED
10

Every file (translog, shard locks, segments, JARs, etc.) opened by Elasticsearch counts as a file descriptor. So do network sockets. I would turn this around: if you think there are too many open, have a look at ls -al /proc/<PID>/fd and tell me what you see that is problematic?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.