Elasticsearch Died on me

Elastic Stack Elasticsearch
21

To keep you updated, I am begrudgingly performing a rolling upgraded (accidentally upgraded one so now I am bottled necked into doing them all).

Adding a few nodes seems to have stabilized it (slightly) I will keep you posted. At the current size we went from 3 masters 3 data to 3 masters 6 data. If this was enough to save our cluster I will ping this thread if not you will hear back from me :smiley:

22

Hey Kurtis,

In relation to your specific question - around trying to protect your nodes from dumb queries take a look at the circuit-breaker settings here --> https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html

As David mentioned its very unlikely that a _search would cause any instability on your cluster.

However I have had experience with crazy aggregations causing node stability issues - even on the current version. Particularly when Kibana times out and the user just keeps clicking to rerun the query - when the first, and the second and the third have not completed.

You should keep any eye on the tasks API to see if there are any queries that are taking longer than 30 seconds to complete --> https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html

1 Like
23

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.