# Cluster crash on query **URL:** \ **Category:** Elasticsearch\ **Created:** [December 16, 2012, 11:00am UTC](https://discuss.elastic.co/t/cluster-crash-on-query/10081 "2012-12-16T11:00:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Richo\_Healey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richo_healey/32/2572_2.png) [@Richo\_Healey](https://discuss.elastic.co/u/Richo_Healey)\ **Post date:** [December 16, 2012, 11:00am UTC](https://discuss.elastic.co/t/cluster-crash-on-query/10081/1 "2012-12-16T11:00:47Z")
We had a 0.19.11 cluster in production for a few weeks. One of our devs rolled out some new code and the cluster bascially came down on itself hard, first complaining of missing shards, and then beginning to ostensibly work when I began manually reindexing data. No migrations were included in the deploy, only some new queries (Which unfortunately I don't have with me now, but I can add tomorrow) The logs from the three nodes are at: [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log) [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log) [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log) Any insight would be welcomed. I've replaced the cluster with one running 0.20.1 for now but really have no guarantees the underlying issue is solved. Richo -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 17, 2012, 12:45pm UTC](https://discuss.elastic.co/t/cluster-crash-on-query/10081/2 "2012-12-17T12:45:50Z")
Hello Richo, From the logs I think your nodes just got too busy and nodes couldn't see each other because of that. So I think you need to either: - optimize the performance on what you already have (if that's possible) - add more nodes - use bigger nodes I think that in order to provide more help, one would need some more information, like: - how many nodes you have, and how big they are? - what's the ES configuration, especially around discovery? - how many indices and shards you have? how much data is in them? how does the data look like (mapping)? - how do the new queries look like? ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Sun, Dec 16, 2012 at 1:00 PM, Richo Healey [healey.rich@gmail.com](mailto:healey.rich@gmail.com) wrote: > We had a 0.19.11 cluster in production for a few weeks. One of our devs > rolled out some new code and the cluster bascially came down on itself > hard, first complaining of missing shards, and then beginning to ostensibly > work when I began manually reindexing data. > > No migrations were included in the deploy, only some new queries (Which > unfortunately I don't have with me now, but I can add tomorrow) > > The logs from the three nodes are at: > > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log) > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log) > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log) > > Any insight would be welcomed. I've replaced the cluster with one running > 0.20.1 for now but really have no guarantees the underlying issue is solved. > > Richo > > -- -- ---
**Author:** ![Richo\_Healey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richo_healey/32/2572_2.png) [@Richo\_Healey](https://discuss.elastic.co/u/Richo_Healey)\ **Post date:** [December 18, 2012, 2:12am UTC](https://discuss.elastic.co/t/cluster-crash-on-query/10081/3 "2012-12-18T02:12:42Z")
Hi Radu, I can certainly add more/larger nodes, but doing so seems like an issue, from what I can see the cluster didn't fall over under load, it actually seems to have lost track of it's shards. I lit up an old cluster this morning, and totally idle sent it the query that broke this one. It immediately went red and started throwing the same errors. The cluster in question is 3x m1.large's, using aws plugin for discovery, fetching nodes in the Security Group (there are only 3) 5 shards, 2 replicas. Only one index I'll dump the mapping and the query in a moment, there are about 180k records. On Monday, 17 December 2012 23:45:50 UTC+11, Radu Gheorghe wrote: > Hello Richo, > > From the logs I think your nodes just got too busy and nodes couldn't see > each other because of that. So I think you need to either: > > - optimize the performance on what you already have (if that's possible) > - add more nodes > - use bigger nodes > > I think that in order to provide more help, one would need some more > information, like: > > - how many nodes you have, and how big they are? > - what's the ES configuration, especially around discovery? > - how many indices and shards you have? how much data is in them? how does > the data look like (mapping)? > - how do the new queries look like? > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Sun, Dec 16, 2012 at 1:00 PM, Richo Healey \<[heale...@gmail.com](mailto:heale...@gmail.com)\ > > > wrote: > > > We had a 0.19.11 cluster in production for a few weeks. One of our devs > > rolled out some new code and the cluster bascially came down on itself > > hard, first complaining of missing shards, and then beginning to ostensibly > > work when I began manually reindexing data. > > > > No migrations were included in the deploy, only some new queries (Which > > unfortunately I don't have with me now, but I can add tomorrow) > > > > The logs from the three nodes are at: > > > > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-29-24-75.log) > > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-64-38-196.log) > > [https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log](https://s3.amazonaws.com/99designs-elasticsearch-logs/ip-10-85-75-6.log) > > > > Any insight would be welcomed. I've replaced the cluster with one running > > 0.20.1 for now but really have no guarantees the underlying issue is solved. > > > > Richo > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:59am UTC](https://discuss.elastic.co/t/cluster-crash-on-query/10081/4 "2017-07-06T02:59:30Z")
Closed on July 6, 2017, 2:59am UTC