# "too many open files" problems and suggestions on cluster configuration **URL:** \ **Category:** Elasticsearch\ **Created:** [May 1, 2015, 3:05pm UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513 "2015-05-01T15:05:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ann\_Yablunovskaya](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@Ann\_Yablunovskaya](https://discuss.elastic.co/u/Ann_Yablunovskaya)\ **Post date:** [May 1, 2015, 3:05pm UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513/1 "2015-05-01T15:05:03Z")
I am looking for suggestions on cluster configuration. I have 2 nodes (master/data and data), 544 indices, about 800 mil documents. If I try to insert more documents and create more indices, I will catch error "too many open files". My node's configuration: CentOS 7 Intel(R) Xeon(R) CPU x16 RAM 62 Gb # ulimit -n 100000 In future I will have a lot of indices (about 2000) and a lot of documents (~5 bil or maybe more) How can I avoid the error "too many open files"? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 1, 2015, 3:21pm UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513/2 "2015-05-01T15:21:25Z")
Add more nodes or reduce the number of shards per node. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 1 mai 2015 à 17:05, Ann Yablunovskaya [lad.shada@gmail.com](mailto:lad.shada@gmail.com) a écrit : > > I am looking for suggestions on cluster configuration. > > I have 2 nodes (master/data and data), 544 indices, about 800 mil documents. > > If I try to insert more documents and create more indices, I will catch error "too many open files". > > My node's configuration: > > CentOS 7 > Intel(R) Xeon(R) CPU x16 > RAM 62 Gb > > # ulimit -n > > 100000 > > In future I will have a lot of indices (about 2000) and a lot of documents (~5 bil or maybe more) > > How can I avoid the error "too many open files"? > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > \ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54E58499-F862-4427-A765-E72FCBDC8D92%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/54E58499-F862-4427-A765-E72FCBDC8D92%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ann\_Yablunovskaya](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@Ann\_Yablunovskaya](https://discuss.elastic.co/u/Ann_Yablunovskaya)\ **Post date:** [May 1, 2015, 3:26pm UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513/3 "2015-05-01T15:26:06Z")
How to calculate the best amount of shards? пятница, 1 мая 2015 г., 18:21:47 UTC+3 пользователь David Pilato написал: > Add more nodes or reduce the number of shards per node. > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 1 mai 2015 à 17:05, Ann Yablunovskaya \<[lad....@gmail.com](mailto:lad....@gmail.com) \\> > a écrit : > > I am looking for suggestions on cluster configuration. > > I have 2 nodes (master/data and data), 544 indices, about 800 mil > documents. > > If I try to insert more documents and create more indices, I will > catch error "too many open files". > > My node's configuration: > > CentOS 7 > Intel(R) Xeon(R) CPU x16 > RAM 62 Gb > > # ulimit -n > > 100000 > > In future I will have a lot of indices (about 2000) and a lot of documents > (~5 bil or maybe more) > > How can I avoid the error "too many open files"? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > \ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7c2e1952-e718-4563-ac5c-bb92b45b0aa5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c2e1952-e718-4563-ac5c-bb92b45b0aa5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 1, 2015, 3:39pm UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513/4 "2015-05-01T15:39:38Z")
The number of open files does not depend on the number of documents. A shard comes not for free. Each shard can take around ~150 open file descriptors (sockets, segment files) and up to 400-500 if actively being indexed. Take care of number of shards, if you have 5 shards per index, and 2000 indices per node, you would hvae to prepare 10k \* 150 open file descriptors. That is a challenge on a single RHEL 7 system providing 131072 file descriptors by default so you would have to change system limits (cat /proc/sys/fs/file-max) - the default is already very high. I recommend using fewer shards and redesign the application for fewer indices (or even a single index) if you are limited to 2 nodes only. You can look at shard routing and index aliasing if this helps: > **[Kagillion Shards | Elasticsearch: The Definitive Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/master/kagillion-shards.html)** > **[Faking Index per User with Aliases | Elasticsearch: The Definitive Guide...](https://www.elastic.co/guide/en/elasticsearch/guide/master/faking-it.html)** Jörg On Fri, May 1, 2015 at 5:05 PM, Ann Yablunovskaya [lad.shada@gmail.com](mailto:lad.shada@gmail.com) wrote: > I am looking for suggestions on cluster configuration. > > I have 2 nodes (master/data and data), 544 indices, about 800 mil > documents. > > If I try to insert more documents and create more indices, I will > catch error "too many open files". > > My node's configuration: > > CentOS 7 > Intel(R) Xeon(R) CPU x16 > RAM 62 Gb > > # ulimit -n > > 100000 > > In future I will have a lot of indices (about 2000) and a lot of documents > (~5 bil or maybe more) > > How can I avoid the error "too many open files"? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_EjkMHgT\_M\_KPvV%3DDSdf-NyidqOziZvg5HXizx8J8rQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_EjkMHgT_M_KPvV%3DDSdf-NyidqOziZvg5HXizx8J8rQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:16am UTC](https://discuss.elastic.co/t/too-many-open-files-problems-and-suggestions-on-cluster-configuration/23513/5 "2017-07-06T00:16:15Z")