# Elasticsearch Died on me

**URL:** <https://discuss.elastic.co/t/elasticsearch-died-on-me/134093>\
**Category:** Elasticsearch\
**Created:** [May 31, 2018, 6:13pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093 "2018-05-31T18:13:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [May 31, 2018, 6:13pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/1 "2018-05-31T18:13:31Z")

</div>

Last night during some document processing, a MASSIVE query was run that would have returned something like 30GB worth of data. As a result, the nodes started to die.

Each machine has 30GB of JVM and 60GB total of RAM.

Is there a way to prevent Elasticsearch from killing itself? If it is working on a query that is going to cause an OOM exception how can I get it to abandon the query instead of committing suicide?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 31, 2018, 6:47pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/2 "2018-05-31T18:47:40Z")

</div>

What is the version?  
What was the query?

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [May 31, 2018, 9:28pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/3 "2018-05-31T21:28:28Z")

</div>

6.2

The query was effectively

GET index/doctype/\_search  
{  
query on some field that returns 30GB worth of data  
}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 2:54am UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/4 "2018-06-01T02:54:17Z")

</div>

> query on some field that returns 30GB worth of data

That's exactly what I'd like to know. Could you tell please what was it?

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 3:56pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/5 "2018-06-01T15:56:44Z")

</div>

```
GET index/doc_type/_search
{
    "query": {
        "range" : {
            "datefield" : {
                "gte" : "2018-05-15T15:02:54.197980"
            }
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 6:35pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/6 "2018-06-01T18:35:18Z")

</div>

I don't think that kind of query can OOM a node. Are you totally sure it's caused by this?

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 6:45pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/7 "2018-06-01T18:45:09Z")

</div>

I am not totally sure, I just saw that at almost the same time that query was being run the node died.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 7:07pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/8 "2018-06-01T19:07:52Z")

</div>

It's probably something else which makes your node dying.  
Do you monitor it with x-pack monitoring?

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 7:09pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/9 "2018-06-01T19:09:35Z")

</div>

I do, the odd thing is that our cluster (its kind of new) has been running without any indication of failure or strain for 2 weeks, then all of the sudden it failed during that query and has been much less stable ever since.

We have added 2 more data nodes, and have had very little improvement. I thought it was the query that was causing the damage, but now I am just lost.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 7:32pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/10 "2018-06-01T19:32:35Z")

</div>

Could you share some details about your cluster?  
Like:

```auto
GET /_cat/health?v
GET /_cat/nodes?v
GET /_cat/fielddata?v
GET /_cat/indices?v

```

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 7:41pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/11 "2018-06-01T19:41:01Z")

</div>

Our cluster is under very heavy load right now and I am reallocating shards off a bad node. but I will dump the info here anyway.

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 7:43pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/12 "2018-06-01T19:43:37Z")

</div>

```
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1527882097 19:41:37 feathrelasticcluster yellow 8 5 4469 2269 4 0 72 0 - 98.4%

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.3.48 1 54 0 0.02 0.03 0.00 m - es-master-2
172.31.2.16 66 99 82 4.15 4.63 5.02 di - es-data-1
172.31.3.158 68 99 61 4.91 5.57 6.45 di - es-data-2
172.31.1.165 10 54 11 0.47 0.40 0.31 m * es-master-3
172.31.2.37 2 54 2 0.20 0.15 0.06 m - es-master-1
172.31.3.242 13 70 8 0.54 0.93 0.94 di - es-data-4
172.31.1.59 46 99 18 1.92 1.29 0.87 di - es-data-5
172.31.1.99 52 99 62 4.72 4.92 4.96 di - es-data-3

```

\

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 7:44pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/13 "2018-06-01T19:44:13Z")

</div>

we have ~500 indexes so pasting this in here is not gonna fit.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 8:00pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/14 "2018-06-01T20:00:07Z")

</div>

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

> **[How many shards should I have in my Elasticsearch cluster?
	  	 | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...

> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 8:06pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/15 "2018-06-01T20:06:23Z")

</div>

I will certainly consider it, but it seems odd that the cluster was fully operational and operating well within reasonable limits, for several weeks (in pre production state) and 2 weeks in full production state. There hasnt been any changes, but it suddenly started failing.

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 8:15pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/16 "2018-06-01T20:15:53Z")

</div>

Also one more question if I can snag you while youre still here 😃

Is there a "rebalance shards" API I can trigger? I have looked through quite a few of the shard allocation api endpoints / settings, but it doesnt seem that there is a "Balance" type command. I added a few nodes, and they did not take up and proportional amount of shards. so 90% of the shards are sitting on the first 3 data nodes (that were the original 3 nodes)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 8:21pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/17 "2018-06-01T20:21:46Z")

</div>

> [@kurtiskurtis](#):
>
> There hasnt been any changes, but it suddenly started failing.

Sure? Did you create new indices, index new documents in the last 2 weeks?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 8:26pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/18 "2018-06-01T20:26:41Z")

</div>

Well. I would not touch at the default settings. Unless you don't have enough disk space on a specific node or using specific [allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-filtering.html), everything should be nicely balanced.

---

<div class="post-metadata">

**Author:** ![kurtiskurtis](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kurtiskurtis](https://discuss.elastic.co/u/kurtiskurtis)\
**Post date:** [June 1, 2018, 8:38pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/19 "2018-06-01T20:38:00Z")

</div>

Well.. let me correct myself... We index a TON of documents per day. nearly 3m a day.

But we have not created any new indexes. So I guess our document count has changed, but 3million is a drop in the bucket compared to our overall dataset.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 1, 2018, 8:59pm UTC](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093/20 "2018-06-01T20:59:30Z")

</div>

What kind of data is that?  
3m per days might be enough to add some pressure on your nodes. Specifically if you are using fielddata instead of doc values and are aggregating tons of different values.

[Next page](https://discuss.elastic.co/t/elasticsearch-died-on-me/134093.md?page=2)