# Unexpected "too many FDs" error on local ES cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [May 16, 2019, 6:07am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310 "2019-05-16T06:07:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 16, 2019, 6:07am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/1 "2019-05-16T06:07:49Z")
I have elasticsearch 6.4.2 running on Mac, inside a container (Docker Desktop). I have about 240 indices and 1100 shards with 0 replicas set. I probably have 100k docs at max (can't say exact count because the cluster is in Red because of FDs error) I see errors in ES logs that says too many open files, when I check `lsof | grep elasticsearch | wc -l` it's giving about 800k+ FDs. This seems to be strange, considering that number of indices isn't large and data is also small. I have seen bigger clusters(in terms of indices/dac count) with far few FDs, I am not sure wha's the issue with my local ES cluster. **ES Data directory stats** 5200 directories, 33015 files. Majority of these files are translog files, I am not sure if this is normal or not. **Node stats endpoints gives this about FDs** { "nodes": { "pSuUtbN-T4e0Ovf03sUS6g": { "process": { "max\_file\_descriptors": 655366 } } } } **ulimit -a** core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 47843 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 655366 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 16, 2019, 6:58am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/2 "2019-05-16T06:58:04Z")
Why do you have so many indices and shards for so little data? It does indeed look strange though. How did you index the data? ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 16, 2019, 8:28am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/3 "2019-05-16T08:28:16Z")
Hi, this is my local cluster and hence some testing which led to those many indices. Yes, I am aware of the issues/overhead with high number of shards. Data is indexed using the bulk APIs, using elasticsearch-js I am also seeing a similar issue on a VM with 100 indices with 200-300k FDs What's strange is that, half of this 200k belong to files related to translog. For ex EDfVKTP2Tni0mrm6K0RrTA doesn't have any docs, but has 1700 open FDs test\_20190422122112\_6l7qj\_7f6bpg7p2v\_data\_ocr5v0\_launchpad\_data\_customer\_viz\_ns\_default EDfVKTP2Tni0mrm6K0RrTA 5 1 0 0 1.2kb 1.2kb > ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 16, 2019, 9:29am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/4 "2019-05-16T09:29:06Z")
I think there is some confusion about what `lsof` is showing. The output here is 1700 lines long but only shows 10 distinct file descriptors in use: ```nohighlight $ cat fds.txt | awk '{ print $5 }' | sort -u 320r 325u 330r 331u 333u 334r 336r 338r 341u 342u REG ``` Each file descriptor is listed multiple times, once for each of the 170 threads. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 16, 2019, 10:01am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/5 "2019-05-16T10:01:33Z")
@DavidTurner Thanks. I didn't know about the per thread entry there. But, I haven't noticed such a huge number even if you account for duplicates in other clusters. Also, this cluster is completely idle when I took these stats about FD, I wonder why there are so many threads running. Also, I have deleted few indices and updated by dev cluster to 6.7.2 and the cluster has started without any issues. Perhaps, even though the FD limit in the container is high, limits on MacOS and hyperkit vm are low. I will update if I run into FD error again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 16, 2019, 10:47am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/6 "2019-05-16T10:47:39Z")
> [@vangap](#): > > Also, this cluster is completely idle when I took these stats about FD, I wonder why there are so many threads running. Elasticsearch uses thread pools to avoid the overhead of creating a new thread each time it's needed. It's possible that the JVM does something similar too. Keeping idle threads alive is normally pretty cheap so 170 threads isn't too surprising. You might get more useful information from these APIs: ```auto GET _nodes/hot_threads?ignore_idle_threads=false GET _nodes/stats/thread_pool ``` ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 30, 2019, 11:30am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/7 "2019-05-30T11:30:47Z")
Hi, I have removed the hyperkit, docker variables now. I have ES running on mac installed via brew. I still see the error Error: > No Of Indices: 98 (I had 2x of these indices which I deleted, restarted) No Of Shards: 473 Host Mac OS ulimit ```auto core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 440000 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2048 virtual memory (kbytes, -v) unlimited ``` `launchctl limit` shows ```auto cpu unlimited unlimited filesize unlimited unlimited data unlimited unlimited stack 8388608 67104768 core 0 unlimited rss unlimited unlimited memlock unlimited unlimited maxproc 2048 2048 maxfiles 440000 524288 ``` Is it possible that the error is misleading and the actual issue might with some other limit? ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 30, 2019, 12:54pm UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/8 "2019-05-30T12:54:26Z")
I did some more testing.. I have `sysctl kern.num_files` around 3k+ when I don't have ES running. It goes up to 13k+ with ES running, ES throws the errors shared above. I am not sure if there is a transient state where this number goes high, I did observe few times and it's always 13k+ afaics. I have decreased the limit to 10k by doing `sudo launchctl limit maxfiles 10000`, I got the error in ES logs that it needs to be atleast 10240. So the limits that I am setting seem to work definitely. This makes me wonder if there is a transient state where it goes to pretty high and then comes back? or the error message is misleading and I might be running into some other related limit ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 30, 2019, 3:08pm UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/9 "2019-05-30T15:08:44Z")
> [@vangap](#): > > Is it possible that the error is misleading and the actual issue might with some other limit? I don't think so. From what I can see, `Too many open files` maps to the error code `EMFILE` ... > ... and the [docs for this error code](http://man7.org/linux/man-pages/man2/open.2.html) don't give any alternative explanations: > **EMFILE** The per-process limit on the number of open file descriptors has been reached (see the description of **RLIMIT\_NOFILE** in [getrlimit(2)](http://man7.org/linux/man-pages/man2/getrlimit.2.html)). Can you share your cluster's settings (`GET _cluster/settings`) and also your `elasticsearch.yml` file(s)? ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 30, 2019, 4:13pm UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/10 "2019-05-30T16:13:43Z")
This is a fresh installation that I just did and copied the data from the docker setup where I ran into this first. So, nothing fancy in my setup. But, I have attached these to the gist > ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 31, 2019, 10:52am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/11 "2019-05-31T10:52:46Z")
Btw, same issue is happening on other members of my team who are using macs and have the number of indices that I have more or less. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 31, 2019, 11:24am UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/12 "2019-05-31T11:24:26Z")
Sorry, debugging this kind of issue on a Mac is outside what I can really help with. I know there are OS-level tools like `dtrace` that might help work out whether Elasticsearch really is opening far too many files or whether there's some other limit that we don't know about, but I have no experience with using them. Can you reproduce this on a [supported OS](https://www.elastic.co/support/matrix)? ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 31, 2019, 12:35pm UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/13 "2019-05-31T12:35:15Z")
@DavidTurner thanks, that's understandable. As mentioned in by original post, this is definitely something to do with Mac/+ES. I will see if any tracing tools might be helpful to narrow it down. I was also trying to find a way to simulate this scenario to see if my system's kern.num\_files goes beyond 13/14k. If it is, it means that OS can handle more FDs, which might suggest that there could be a transient state where ES might be having more FDs is a likely problem. If you know of any way to simulate this, pls let me know. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 28, 2019, 12:35pm UTC](https://discuss.elastic.co/t/unexpected-too-many-fds-error-on-local-es-cluster/181310/14 "2019-06-28T12:35:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.