# Rollup - date histogram issues

**URL:** <https://discuss.elastic.co/t/rollup-date-histogram-issues/328509>\
**Category:** Kibana\
**Tags:** lens, rollups\
**Created:** [March 25, 2023, 8:26am UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509 "2023-03-25T08:26:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![JeroenK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeroenk/32/103640_2.png) [@JeroenK](https://discuss.elastic.co/u/JeroenK)\
**Post date:** [March 25, 2023, 8:26am UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/1 "2023-03-25T08:26:38Z")

</div>

I have a rollup job with the following settings:

```auto
          "groups": {
            "date_histogram": {
              "field": "timestamp",
              "time_zone": "Europe/Stockholm",
              "calendar_interval": "1d"
            },

```

I also tried ` "calendar_interval": "24h"` btw

As I understand from the documentation about rollups, using a smaller date histogram interval is not possible, but using a bigger date histogram is. Therefore, I'm expecting when I create a Lens visualisation, I will be able to choose 1d, 2d, 1w, 1M, etc. when the `"calendar_interval": "1d"` and similar scenarios with the 24h setting (48h for example)

What I'm getting in Lens is the following error

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3e9fb444fa9c2188f78ed215d268cea320deeb31.png)

I've tried other visualisations too, nothing really works

What I want to achieve is having daily totals of our transaction data and building dashboards with date histograms with intervals of 1d, 1w, 1M and 1y (ideally all calendar based since reporting in this business is calendar based).

Hopefully someone can point me in the right direction on how to achieve this?

Jeroen

---

<div class="post-metadata">

**Author:** ![JeroenK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeroenk/32/103640_2.png) [@JeroenK](https://discuss.elastic.co/u/JeroenK)\
**Post date:** [March 25, 2023, 8:40am UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/2 "2023-03-25T08:40:03Z")

</div>

Funny enough, this works fine:

```auto
POST /lp-reporting-transactions-daily/_search?size=0
{
  "aggs": {
    "transactions": {
      "date_histogram": {
        "field": "timestamp.date_histogram.timestamp",
        "calendar_interval": "month"
      }
    }
  }
}

```

same for `week` and `year`

It feels like the issue is related to Lens and other visualisations?

---

<div class="post-metadata">

**Author:** ![drewdaemon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewdaemon/32/97779_2.png) [@drewdaemon](https://discuss.elastic.co/u/drewdaemon)\
**Post date:** [April 12, 2023, 9:31pm UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/3 "2023-04-12T21:31:32Z")

</div>

Hi @JeroenK, Rollups are in technical preview and may not be around forever. The support in Kibana (Lens included) is limited.

The [time series data stream](https://www.elastic.co/guide/en/elasticsearch//reference/master/tsds.html) with [downsampling](https://www.elastic.co/guide/en/elasticsearch/reference/master/downsampling.html) is a "blessed" solution moving forward. Does it look like it would fit your needs?

---

<div class="post-metadata">

**Author:** ![neoaddix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neoaddix/32/118260_2.png) [@neoaddix](https://discuss.elastic.co/u/neoaddix)\
**Post date:** [April 13, 2023, 7:53am UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/4 "2023-04-13T07:53:04Z")

</div>

Thanks @drewdaemon that explains things

I was looking at downsampling indeed, but what I could not figure out is if you loose any data? It reads like it 'samples' from the data instead of actually rolling it up.

My case is for analytical data about transactions and I want to aggregate what we have to daily totals. Will downsampling do that?

---

<div class="post-metadata">

**Author:** ![drewdaemon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewdaemon/32/97779_2.png) [@drewdaemon](https://discuss.elastic.co/u/drewdaemon)\
**Post date:** [April 13, 2023, 5:19pm UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/5 "2023-04-13T17:19:20Z")

</div>

@neoaddix , with both rollups and TSDS downsampling you're taking a highly-sampled index (lots of documents) and transforming it into an index with fewer documents each of which summarizes some number of documents from the original index. So, really they're both "lossy compression" techniques and they could both be called "downsampling."

Each of the documents in the downsampled index stores aggregation information for the documents from the original index that it represents. In the case of TSDS downsampling, we store the `min` , `max` , `sum` , `value_count` , and `average` for each metric. (IIRC, rollups allow you to specify which of these you retain.)

The object is to reduce storage costs while improving performance for certain aggregations (fewer documents to aggregate = faster).

You can imagine it like compressing an image where every four pixels becomes one single pixel, the average of the original four. When the image is at its normal size after compression, you probably won't notice a difference. However, the compression did reduce your ability to zoom in and see more granular details.

(Sorry if you knew all this, just stating it "out-loud.")

> [@neoaddix](#):
>
> My case is for analytical data about transactions and I want to aggregate what we have to daily totals. Will downsampling do that?

Full disclosure: this is a little outside my area of expertise—might want to verify this in the Elasticsearch topic.

That said, I think this is probably the perfect case for downsampling. As you can see in [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/downsampling.html#running-downsampling), you can downsample with an interval of one day `"fixed_interval": "1d"` (so in your downsampled index, you get one document summarizing each day). And you can do this with an ILM rule so that it happens automatically.

You could always run a downsample using the API first to make sure things look as they should. Just remember that

> Within a data stream, a downsampled index replaces the original index and the original index is deleted. Only one index can exist for a given time period.

so, when you run the downsample command, the original documents are gone (from [doc](https://www.elastic.co/guide/en/elasticsearch/reference/master/downsampling.html#downsampling-restrictions)).

Does this help?

---

<div class="post-metadata">

**Author:** ![neoaddix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neoaddix/32/118260_2.png) [@neoaddix](https://discuss.elastic.co/u/neoaddix)\
**Post date:** [April 14, 2023, 3:47pm UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/6 "2023-04-14T15:47:42Z")

</div>

Thanks @drewdaemon this is very helpful, I will play with this to validate if it covers what I need. I'll post my results here

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 12, 2023, 3:48pm UTC](https://discuss.elastic.co/t/rollup-date-histogram-issues/328509/7 "2023-05-12T15:48:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.