Rollup index

[SNJY]

Hi - I am trying to test feature of Rollup Index in ES6.6. From kibana, I created rollup job of metricbeat data based on time series @timestamp.
While adding rollup index in grafana data source, there is error message "No date field named @timestamp found". I am curious to know if someone has implemented rollup index with grafana.
Also - could someone please share any webinar recording on rollup ( if any ). Else can a webinar be arranged on rollup.
Thanks,
Sanjay

[BenTrent]

@SNJY

Rollup Indices store only what you specify. Additionally, the fields that are actually stored inside each document are specific to the rollup job.

So, take kibana_sample_data_flights data set for example:

Rollup Job Config of:

PUT _xpack/rollup/job/flight_rollup
{
    "index_pattern": "kibana_sample_data_flights",
    "rollup_index": "kibana_sample_data_flights_rollup",
    "cron": "*/30 * * * * ?",
    "page_size" :1000,
    "groups" : {
      "date_histogram": {
        "field": "timestamp",
        "interval": "1d"
      },
      "terms": {
        "fields": ["OriginAirportID", "DestAirportID", "DestWeather", "OriginWeather"]
      }
    },
    "metrics": [
        {
            "field": "FlightTimeMin",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "AvgTicketPrice",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "DistanceMiles",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
            "field": "FlightDelayMin",
            "metrics": ["min", "max", "sum", "avg"]
        },
        {
          "field": "timestamp",
          "metrics": ["min", "max"]
        }
    ]
}

Will result in documents like this:

"_source" : {
          "AvgTicketPrice.sum.value" : 655.3579711914062,
          "FlightTimeMin.min.value" : 1138.500732421875,
          "OriginWeather.terms._count" : 1,
          "DistanceMiles.avg.value" : 9904.0419921875,
          "FlightTimeMin.avg._count" : 1.0,
         ...,
          "timestamp.date_histogram.timestamp" : 1544400000000,
          "timestamp.date_histogram._count" : 1,
          ...
        }

To actually do searches and aggregations against rollup indices, you need to use the specified _rollup_search which translates the query to match the actually stored fields, and then translates the results so that they are intelligible.

Taking a quick look at the elastic search data source code in grafana, there are a couple of places (at least) where code would need to change to support rollup indices:

• Determining the data mapping and supported fields As this will look at the rollup index mapping directly and not see the rolled up fields. It should probably use something like: rollup-get-rollup-caps
• Using _msearch for querying instead of _rollup_search. Using _rollup_search is necessary when querying rolled up indices for data

Searching the open issues on Grafana turned up this opened issue: https://github.com/grafana/grafana/issues/12267