# Rollup index

**URL:** <https://discuss.elastic.co/t/rollup-index/173804>\
**Category:** Elasticsearch\
**Created:** [March 25, 2019, 5:35pm UTC](https://discuss.elastic.co/t/rollup-index/173804 "2019-03-25T17:35:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SNJY](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/snjy/32/42758_2.png) [@SNJY](https://discuss.elastic.co/u/SNJY)\
**Post date:** [March 25, 2019, 5:35pm UTC](https://discuss.elastic.co/t/rollup-index/173804/1 "2019-03-25T17:35:02Z")

</div>

Hi - I am trying to test feature of Rollup Index in ES6.6. From kibana, I created rollup job of metricbeat data based on time series @timestamp.  
While adding rollup index in grafana data source, there is error message "No date field named @timestamp found". I am curious to know if someone has implemented rollup index with grafana.  
Also - could someone please share any webinar recording on rollup ( if any ). Else can a webinar be arranged on rollup.  
Thanks,  
Sanjay

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [March 25, 2019, 7:23pm UTC](https://discuss.elastic.co/t/rollup-index/173804/2 "2019-03-25T19:23:35Z")

</div>

@SNJY

Rollup Indices store only what you specify. Additionally, the fields that are actually stored inside each document are specific to the rollup job.

So, take `kibana_sample_data_flights` data set for example:

Rollup Job Config of:

```auto
PUT _xpack/rollup/job/flight_rollup
{
    "index_pattern": "kibana_sample_data_flights",
    "rollup_index": "kibana_sample_data_flights_rollup",
    "cron": "*/30 * * * * ?",
    "page_size" :1000,
    "groups" : {
      "date_histogram": {
        "field": "timestamp",
        "interval": "1d"
      },
      "terms": {
        "fields": ["OriginAirportID", "DestAirportID", "DestWeather", "OriginWeather"]
      }
    },
    "metrics": [
        {
            "field": "FlightTimeMin",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "AvgTicketPrice",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "DistanceMiles",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "FlightDelayMin",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
          "field": "timestamp",
          "metrics": ["min", "max"]
        }
    ]
}

```

Will result in documents like this:

```auto
"_source" : {
          "AvgTicketPrice.sum.value" : 655.3579711914062,
          "FlightTimeMin.min.value" : 1138.500732421875,
          "OriginWeather.terms._count" : 1,
          "DistanceMiles.avg.value" : 9904.0419921875,
          "FlightTimeMin.avg._count" : 1.0,
         ...,
          "timestamp.date_histogram.timestamp" : 1544400000000,
          "timestamp.date_histogram._count" : 1,
          ...
        }

```

To actually do searches and aggregations against rollup indices, you need to use the specified `_rollup_search` which translates the query to match the actually stored fields, and then translates the results so that they are intelligible.

Taking a quick look at the elastic search data source code in grafana, there are a couple of places (at least) where code would need to change to support rollup indices:

- [Determining the data mapping and supported fields](https://github.com/grafana/grafana/blob/4c42db9e95b755bbd6f24c302ab65ce04fe58b7a/public/app/plugins/datasource/elasticsearch/datasource.ts#L286) As this will look at the rollup index mapping directly and not see the rolled up fields. It should probably use something like: [rollup-get-rollup-caps](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/rollup-get-rollup-caps.html)
- [Using \_msearch for querying instead of \_rollup\_search](https://github.com/grafana/grafana/blob/4c42db9e95b755bbd6f24c302ab65ce04fe58b7a/public/app/plugins/datasource/elasticsearch/datasource.ts#L281). Using [\_rollup\_search](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/rollup-search.html) is necessary when querying rolled up indices for data

Searching the open issues on Grafana turned up this opened issue: [https://github.com/grafana/grafana/issues/12267](https://github.com/grafana/grafana/issues/12267)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2019, 7:35pm UTC](https://discuss.elastic.co/t/rollup-index/173804/3 "2019-04-22T19:35:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.