# ES 2.4 to 5.2 Upgrade Followed By Major Cluster Instability

**URL:** <https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943>\
**Category:** Elasticsearch\
**Created:** [March 24, 2017, 7:45pm UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943 "2017-03-24T19:45:40Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\
**Post date:** [March 28, 2017, 9:56pm UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943/21 "2017-03-28T21:56:29Z")

</div>

@mstruve - any script usage on that cluster (Painless)?

---

<div class="post-metadata">

**Author:** ![mstruve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mstruve/32/50419_2.png) [@mstruve](https://discuss.elastic.co/u/mstruve)\
**Post date:** [March 28, 2017, 10:39pm UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943/22 "2017-03-28T22:39:03Z")

</div>

@Itamar_Syn_Hershko We use scripts in two places:

1. One expression script when we compile reports nightly it is a single call and we use it to calculate the time between two fields and aggregate on that.
2. We use a single painless script on occasion to allow users to update\_by\_query some fields. This is maybe used once or twice a week and there is a good chance it hasn't even been used on the new cluster by anyone.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 29, 2017, 5:10am UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943/23 "2017-03-29T05:10:59Z")

</div>

Please start your own thread . for this 🙂

---

<div class="post-metadata">

**Author:** ![mstruve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mstruve/32/50419_2.png) [@mstruve](https://discuss.elastic.co/u/mstruve)\
**Post date:** [March 29, 2017, 10:49pm UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943/24 "2017-03-29T22:49:17Z")

</div>

OK folks, it looks like we have found a solution for our stability issue.

After sending ES a heap dump from one of our spikes they were able to [find a bug](https://github.com/elastic/elasticsearch/pull/23797) that affects filters in aggregations. In filtered aggs like the one below the bug causes the filter to be run as a bool query and it is therefore scored. This is a big issue for us bc we make A LOT of these.

```auto
{
  "agg_name":{
    "filter":{
      "terms":{
        "id":[1,2,...]
      }
    }
  }
}

```

The temp fix is to use constant\_score like this to avoid having the filter scored.

```auto
{ 
  "agg_name": {
    "filter": {
     "constant_score": {
       "filter": {
         "terms": {
            "id": []
          }
        }
      }
   }
}

```

That change looks to have completely stabilized us. The change went out at noon and this is how the graphs look pre and post change. Healthy saw tooth. We will be monitoring it closely and repost if anything changes. Thanks everyone for the help!

 ![](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a515fdc6ee48e0e712d4c0f7408c88995dacd034.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2017, 10:49pm UTC](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943/25 "2017-04-26T22:49:33Z")

</div>

Closed on April 26, 2017, 10:49pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/es-2-4-to-5-2-upgrade-followed-by-major-cluster-instability/79943.md?page=1)