# Slow date\_histogram after upgrading to 7.3.0 on "dense" indexes

**URL:** <https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475>\
**Category:** Elasticsearch\
**Created:** [August 23, 2019, 9:15am UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475 "2019-08-23T09:15:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [August 23, 2019, 9:15am UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/1 "2019-08-23T09:15:10Z")

</div>

Hi there,

We noticed a serious performance degradation of `date_histogram` aggregations on our cluster after migrating from ES `6.8.2` to `7.3.0`. Queries are at least 2 times slower with `7.3.0`.

We managed to reproduce the issue in a test environment and compared results between the two versions. For this test we made use of 4 distinct datasets, each made of `70m` documents containing a single `@timestamp` field:

- `gaussian-sameday`: this dataset represents the actual distribution of log data during a production day. It is a gaussian distribution centered around lunch time (more documents during the day than the night). All documents fit within the same day.
- `uniform-sameday`: All documents fit within the same day but are evenly distributed (same amount of docs every hours).
- `uniform-1s`: Documents are spaced a second apart (the first starts at `2000-01-01T00:00:00.000Z`, next is 1 second later).
- `uniform-10s`: 10 second gap between documents.

Each dataset is loaded in its own index configured with a single shard and no replica. The test cluster is made of a single ES node with the default configuration as shipped by Elastic.

We ran the following query against the 4 datasets with query caching disabled:

```auto
{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "1d",
        "min_doc_count": 1
      }
    }
  },
  "query": {
    "match_all": {}
  }
}

```

The table below shows the timing in millis:

```auto
dataset 6.8.2 7.3.0
--------------------------------------
gaussian-sameday 6520 11450
uniform-sameday 6280 12013
uniform-1s 34836 16199
uniform-10s 34211 16934

```

**As you noticed, `6.8.2` performs twice faster than `7.3.0` when dates are _close to each other_ (_sameday_ datasets). However, `7.3.0` performs better when they are spread over a larger time range.**

We ran the same queries with different bucket sizes and concluded it has no or little impact as we always observed the same differences between the two versions.

Did someone experienced the same behaviour?  
Is this something we can tune?  
Is this a regression?

Thanks for your advices,  
/Bertrand

Note: we could not reproduce the issue with the `nyc_taxis` dataset used by Rally because dates in that dataset are not close enough to each other.

* * *

Update: removing `time_zone` parameter from the `date_histogram` since it is not required by the scenario. Timings are obviously different now but differences between the two ES versions are still the same.

---

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [August 26, 2019, 12:36pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/2 "2019-08-26T12:36:51Z")

</div>

Any feedback from Elastic members on this ?  
For the info, we got same results with `7.3.1`.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 26, 2019, 1:04pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/3 "2019-08-26T13:04:35Z")

</div>

Hey,

can you open a github issue for this for further investigation, please?

--Alex

---

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [August 26, 2019, 1:21pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/4 "2019-08-26T13:21:44Z")

</div>

@spinscale Maybe you could reopen [https://github.com/elastic/elasticsearch/issues/45702](https://github.com/elastic/elasticsearch/issues/45702) or do you want me to create a new one by copying this forum post ?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 26, 2019, 1:58pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/5 "2019-08-26T13:58:48Z")

</div>

I reopened. If you have any possibilities to share your dataset or any further information to put on that issue, that would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [August 26, 2019, 2:00pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/6 "2019-08-26T14:00:54Z")

</div>

Datasets are all about `50m` bzip2 compressed files. Where should I put them? In a temporary public github repo?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 26, 2019, 2:03pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/7 "2019-08-26T14:03:26Z")

</div>

maybe upload to gdrive and then provide a link? If you cannot share publicly, but are willing to share privately, you can email me a private link and I'm happy to forward that internally. My email is $firstname.$lastname@elastic.co\_without\_an\_m\_at\_the\_end

--Alex

---

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [August 26, 2019, 2:09pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/8 "2019-08-26T14:09:28Z")

</div>

The four datasets mentioned in my test cases are available at [https://drive.google.com/open?id=1ZIQjh00zOFyLoLY8vTEo2Tm2T0ZJ4AnD](https://drive.google.com/open?id=1ZIQjh00zOFyLoLY8vTEo2Tm2T0ZJ4AnD)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2019, 2:09pm UTC](https://discuss.elastic.co/t/slow-date-histogram-after-upgrading-to-7-3-0-on-dense-indexes/196475/9 "2019-09-23T14:09:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.