# Elasticsearch opening way too many (200K) Pipes, FDs **URL:** \ **Category:** Elasticsearch\ **Created:** [October 30, 2017, 10:51pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817 "2017-10-30T22:51:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![rajeev](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@rajeev](https://discuss.elastic.co/u/rajeev)\ **Post date:** [October 30, 2017, 10:51pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/1 "2017-10-30T22:51:09Z")
We have elasticsearch running in our product, its not a cluster configuration. We have noticed despite maximum FD limit set to 65536, it has opened too many Pipes/ root]# lsof | grep -i elasticsearch | wc -l 204875 root]# Out of these open files, almost 160000 are pipes and inodes. java 1966 elasticsearch 92r FIFO 0,9 0t0 34857 pipe java 1966 elasticsearch 93w FIFO 0,9 0t0 34857 pipe java 1966 elasticsearch 94u a\_inode 0,10 0 1035 [eventpoll] If we look for particular elasticsearch process under /proc//fd we have about 671 open FD. But I can understand why for elastic user so many other FDs are opened in the system \_nodes/stats/process?filter\_path=\*\*.max\_file\_descriptors {}{"nodes":{"s8U48FmKSdGal6JyXuNnzg":{"process":{"max\_file\_descriptors":65535}}}} Can you please help me understand this behavior and how it can be eliminated? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 31, 2017, 4:15am UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/2 "2017-10-31T04:15:47Z")
A first guess is the number of shards per node. How many do you have? ---
**Author:** ![rajeev](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@rajeev](https://discuss.elastic.co/u/rajeev)\ **Post date:** [October 31, 2017, 5:39pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/3 "2017-10-31T17:39:21Z")
Hi David, Its 1 { "DB1\_v2" : { "settings" : { "index" : { "number\_of\_shards" : "1", "max\_result\_window" : "1000000", "creation\_date" : "1509099442192", "analysis" : { "analyzer" : { "lower\_term" : { "filter" : "lowercase", "type" : "custom", "tokenizer" : "keyword" } } }, "number\_of\_replicas" : "1", "uuid" : "jmrPGEK7Twmo6qS6al9A-g", "version" : { "created" : "2030499" } } } }, "DB2\_v1" : { "settings" : { "index" : { "number\_of\_shards" : "1", "max\_result\_window" : "1000000", "creation\_date" : "1509099442386", "analysis" : { "analyzer" : { "lower\_term" : { "filter" : "lowercase", "type" : "custom", "tokenizer" : "keyword" } } }, "number\_of\_replicas" : "1", "uuid" : "48X3eMQzTzidE\_5Q6KY9bw", "version" : { "created" : "2030499" } } } } } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 31, 2017, 6:23pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/4 "2017-10-31T18:23:52Z")
Please format your code using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will make your post more readable. Or use markdown style like: ```` ``` CODE ``` ```` What gives: ``` GET _cat/shards?v ``` ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [October 31, 2017, 7:14pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/5 "2017-10-31T19:14:21Z")
`lsof` as you have done is not an accurate way to count open file descriptors. For example, as you have done will count memory-mapped portions of files which do not count individually as file descriptors. Additionally, `lsof` will duplicate the same file descriptor multiple times for multiple threads. The correct way to check the number of file descriptors is the number of entires in `/proc//fd`. ---
**Author:** ![rajeev](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@rajeev](https://discuss.elastic.co/u/rajeev)\ **Post date:** [October 31, 2017, 7:18pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/7 "2017-10-31T19:18:57Z")
I agree, but on a idle system, why we should have so may open FDs pointed by elasticsearch. There are several other services but elastic is still contributing about 80% of open FDs in system ---
**Author:** ![rajeev](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@rajeev](https://discuss.elastic.co/u/rajeev)\ **Post date:** [October 31, 2017, 7:22pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/8 "2017-10-31T19:22:05Z")
Sorry for spamming just tried one more time ``` GET _settings?pretty { "DB1_v2" : { "settings" : { "index" : { "number_of_shards" : "1", "max_result_window" : "1000000", "creation_date" : "1509099442192", "analysis" : { "analyzer" : { "lower_term" : { "filter" : "lowercase", "type" : "custom", "tokenizer" : "keyword" } } }, "number_of_replicas" : "1", "uuid" : "jmrPGEK7Twmo6qS6al9A-g", "version" : { "created" : "2030499" } } } }, "DB2_v1" : { "settings" : { "index" : { "number_of_shards" : "1", "max_result_window" : "1000000", "creation_date" : "1509099442386", "analysis" : { "analyzer" : { "lower_term" : { "filter" : "lowercase", "type" : "custom", "tokenizer" : "keyword" } } }, "number_of_replicas" : "1", "uuid" : "48X3eMQzTzidE_5Q6KY9bw", "version" : { "created" : "2030499" } } } } } ``` ---
**Author:** ![rajeev](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@rajeev](https://discuss.elastic.co/u/rajeev)\ **Post date:** [October 31, 2017, 7:25pm UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/9 "2017-10-31T19:25:21Z")
``` _cat/shards?v index shard prirep state docs store ip node DB2_v1 0 p STARTED 0 159b 127.0.0.1 Cletus Kasady DB2_v1 0 r UNASSIGNED DB1_v2 0 p STARTED 4686 2.1mb 127.0.0.1 Cletus Kasady DB1_v2 0 r UNASSIGNED ``` ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [November 1, 2017, 1:13am UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/10 "2017-11-01T01:13:58Z")
Every file (translog, shard locks, segments, JARs, etc.) opened by Elasticsearch counts as a file descriptor. So do network sockets. I would turn this around: if you think there are too many open, have a look at `ls -al /proc//fd` and tell me what you see that is problematic? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 29, 2017, 1:14am UTC](https://discuss.elastic.co/t/elasticsearch-opening-way-too-many-200k-pipes-fds/105817/11 "2017-11-29T01:14:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.