# ES Keeps Falling Over **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2012, 4:11pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996 "2012-06-06T16:11:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dan\_Lecocq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_lecocq/32/2565_2.png) [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)\ **Post date:** [June 6, 2012, 4:11pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/1 "2012-06-06T16:11:47Z")
We've really been enjoying elasticsearch, where we were having some unpleasantness with Sphinx. At any rate, we're continuously inserting batches of documents through a job queue, using the python bindings ([https://github.com/aparo/pyes](https://github.com/aparo/pyes)), and after a couple of hours, the cluster reliably falls apart. We've got 6 m1.larges running the thing with 300 shards (we want to have room to grow if need be). I've noticed that it periodically complains about the number of open file descriptors, despite it being set to 64k on each machine. From the logs, it seems that eventually shard recovery fails, and then it goes downhill from there. Restarting the whole cluster brings it back into the green, but I have a feeling that's not meant to be a reguarly-run ops issue. When inserting documents, we're using the bulk API, and then waiting for green status. Rather, that's what we're requesting of the python API. I'm not entirely convinced that there's not something going on in that library, but I've not been able to track anything obvious down in there. Gist with mapping, health, machine configuration: [https://gist.github.com/0c175c2b0ed3d84138b6](https://gist.github.com/0c175c2b0ed3d84138b6) Is this symptomatic of a common problem? Or a known problem? I imagine it's something I've not set up correctly :-/ ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 10, 2012, 9:29pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/2 "2012-06-10T21:29:18Z")
Can you use the nodes info API to check if the file descriptor settings has actually been applied to the process? Node stats will show you the current open file handles. You can use bigdesk to see that visually. Also, why 300 shards? Can you share a bit more your model? See some info here: [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/data$20flow/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ), and here: [https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics](https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics). On Wed, Jun 6, 2012 at 6:11 PM, Dan Lecocq [dan@seomoz.org](mailto:dan@seomoz.org) wrote: > We've really been enjoying elasticsearch, where we were having some > unpleasantness with Sphinx. At any rate, we're continuously inserting > batches of documents through a job queue, using the python bindings ( > [GitHub - aparo/pyes: Python connector for ElasticSearch - the pythonic way to use ElasticSearch](https://github.com/aparo/pyes)), and after a couple of hours, the cluster > reliably falls apart. We've got 6 m1.larges running the thing with 300 > shards (we want to have room to grow if need be). > > I've noticed that it periodically complains about the number of open file > descriptors, despite it being set to 64k on each machine. From the logs, it > seems that eventually shard recovery fails, and then it goes downhill from > there. Restarting the whole cluster brings it back into the green, but I > have a feeling that's not meant to be a reguarly-run ops issue. > > When inserting documents, we're using the bulk API, and then waiting for > green status. Rather, that's what we're requesting of the python API. I'm > not entirely convinced that there's not something going on in that library, > but I've not been able to track anything obvious down in there. > > Gist with mapping, health, machine configuration: > [https://gist.github.com/0c175c2b0ed3d84138b6](https://gist.github.com/0c175c2b0ed3d84138b6) > > Is this symptomatic of a common problem? Or a known problem? I imagine > it's something I've not set up correctly :-/ ---
**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\ **Post date:** [June 10, 2012, 10:56pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/3 "2012-06-10T22:56:53Z")
Yes, I have such experience with the max open file. You'd better check the Java process's max open file by: car /proc/pidofjava/limit|grep file in Linux or something like in other OS. Alan ## Sent from my Windows Phone From: Shay Banon Sent: 6/11/2012 5:29 AM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: ES Keeps Falling Over Can you use the nodes info API to check if the file descriptor settings has actually been applied to the process? Node stats will show you the current open file handles. You can use bigdesk to see that visually. Also, why 300 shards? Can you share a bit more your model? See some info here: [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/data$20flow/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ), and here: [elasticsearch, Big Data, Search & Analytics - Speaker Deck](https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics). On Wed, Jun 6, 2012 at 6:11 PM, Dan Lecocq [dan@seomoz.org](mailto:dan@seomoz.org) wrote: > We've really been enjoying elasticsearch, where we were having some > unpleasantness with Sphinx. At any rate, we're continuously inserting > batches of documents through a job queue, using the python bindings ( > [GitHub - aparo/pyes: Python connector for ElasticSearch - the pythonic way to use ElasticSearch](https://github.com/aparo/pyes)), and after a couple of hours, the cluster > reliably falls apart. We've got 6 m1.larges running the thing with 300 > shards (we want to have room to grow if need be). > > I've noticed that it periodically complains about the number of open file > descriptors, despite it being set to 64k on each machine. From the logs, it > seems that eventually shard recovery fails, and then it goes downhill from > there. Restarting the whole cluster brings it back into the green, but I > have a feeling that's not meant to be a reguarly-run ops issue. > > When inserting documents, we're using the bulk API, and then waiting for > green status. Rather, that's what we're requesting of the python API. I'm > not entirely convinced that there's not something going on in that library, > but I've not been able to track anything obvious down in there. > > Gist with mapping, health, machine configuration: > [https://gist.github.com/0c175c2b0ed3d84138b6](https://gist.github.com/0c175c2b0ed3d84138b6) > > Is this symptomatic of a common problem? Or a known problem? I imagine > it's something I've not set up correctly :-/ ---
**Author:** ![Dan\_Lecocq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_lecocq/32/2565_2.png) [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)\ **Post date:** [June 10, 2012, 11:08pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/4 "2012-06-10T23:08:27Z")
Thanks for replies; @Shay - I've seen my open fd count spike at 64k, and though I've only just started checking the \_nodes/stats endpdoint, since ES is the only thing running on those boxes, I'm relatively certain those fds were owned by ES. WRT the shards, I initially didn't have much of a sense of how many was too many, and wanted to have room to grow, which bring me to our data flow. We're hoping to insert on the order of maybe a few million documents (not saving content) per day, and keeping the last 90 days of history in the index (using \_ttl). @LiMac - I have verified that my 64k limit has been applied to these. It's something I hit earlier, when I had it set to 32k, and had to up it (to no avail, it would seem). - D ---
**Author:** ![Dan\_Lecocq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_lecocq/32/2565_2.png) [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)\ **Post date:** [June 11, 2012, 6:32pm UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/5 "2012-06-11T18:32:03Z")
In light of learning about aliasing, and since our data is very conducive to time-based slicing, I'm now planning on using many fewer shards over more indexes, and then aliasing them as an administrative task. It seems to me that this will also make scaling up easier, so that when I need to add more nodes, I can create a new index with sufficiently many shards, set that as the write index, and then introduce the new nodes? The added flexibility will be nice. On Sunday, June 10, 2012 4:08:27 PM UTC-7, Dan Lecocq wrote: > Thanks for replies; > > @Shay - I've seen my open fd count spike at 64k, and though I've only just > started checking the \_nodes/stats endpdoint, since ES is the only thing > running on those boxes, I'm relatively certain those fds were owned by ES. > WRT the shards, I initially didn't have much of a sense of how many was too > many, and wanted to have room to grow, which bring me to our data flow. > We're hoping to insert on the order of maybe a few million documents (not > saving content) per day, and keeping the last 90 days of history in the > index (using \_ttl). > > @LiMac - I have verified that my 64k limit has been applied to these. It's > something I hit earlier, when I had it set to 32k, and had to up it (to no > avail, it would seem). > > - D ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/es-keeps-falling-over/7996/6 "2017-07-06T03:24:54Z")