Rollup index date_histogram issue

Elastic Stack Elasticsearch
1

Hi,
Getting the following error when trying to Visualize a count on the Y Axis and Date Histogram of type field @timestamp with an interval of 60m on the X AxIs over Today.

Rollup search error: [illegal_argument_exception] There is not a rollup job that has a [date_histogram] agg on field [@timestamp] which also satisfies all requirements of query.

Running elasticsearch 7.0.0
The index pattern I am visualizing against shares both the unrolled metricbeat index and the rollup 60m interval index.

Below is the top part of the json for the rollup job.

     "config": {
        "id": "metricbeat_1h",
        "index_pattern": "metricbeat-*",
        "rollup_index": "rollup_1h_metricbeat",
        "cron": "0 15 * * * ?",
        "groups": {
          "date_histogram": {
            "interval": "60m",
            "field": "@timestamp",
            "delay": "7d",
            "time_zone": "UTC"
          },

I do not have the timestamp field anywhere else in the rollup json as I had believed that the date_histogram would be enough to allow for visualizing using the timestamp field.

There is nothing showing in the elasticsearch logs.

Not sure if it is an issue with Elasticsearch or something I have set wrong.

2

Can you paste the (generated) query that Kibana is trying to execute?

There are basically three components that need to match when running a query:

  • All the fields in the query/aggregation need to be in the job
  • All the parameters are compatible; e.g. if the job was rolled up with 60m, the aggregation needs to use a multiple of 60m. Ditto to Timezones, etc
  • All the requested aggregations were rolled up (averages, histograms, etc).

My first guess is that a parameter is not matching, like the interval or timezone.

3

Think this is the right bit.

{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "1h",
        "time_zone": "UTC",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "max": {
            "field": "system.filesystem.used.bytes"
          }
        }
      }
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2019-04-29T04:28:48.838Z",
              "lte": "2019-04-29T15:58:21.157Z"
            }
          }
        }
      ],
      "filter": [
        {
          "match_all": {}
        },
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}
4

Ahh, I think this is the cause.

Elasticsearch has an unfortunate system for intervals at the moment. You can read more about it here (scroll down to "Calendar vs Fixed Time Intervals"): Rollup job configuration | Elasticsearch Guide [6.7] | Elastic, and at the date histogram docs: Date Histogram Aggregation | Elasticsearch Guide [6.8] | Elastic

Basically, there are a set of units which are "calendar-aware", meaning they understand things like leap seconds, number of days or weeks per month, daylight savings time, etc. These units are singular quantities like 1h, 1d, 1w, 1M, etc.

Then there are "fixed" intervals which are strictly multiples of SI milliseconds. 2m is 1000ms * 60 * 2, etc.

Fixed intervals are not compatible with calendar and vice versa. So what happened is your Rollup job is defined as 60m (fixed time), but the query is asking for 1h (calendar time), and that's where things are going wrong. If you change the query to 60m I think it should work.

This is definitely a very confusing situation, and something that most users don't even know about. We just recently merged a PR to make this distinction explicit with two new fields on the date histogram agg (calendar_interval and fixed_interval), to hopefully clear this confusion up in the future.

5

Thanks will have a read.

Inside Kibana on the visualisation the interval is set to 60m, with the query outputting 1h.

If this is an issue that 7.2.0 kibana will fix. More then happy to wait till then. I have the none rollup version of this visualisation working, just not including the amount of data I was hoping for :slight_smile: .

6

Oh, this is interesting. Might be a bug with how Kibana is generating the queries then. Lemme ping a few folks and direct them to this ticket.

The calendar_interval/fixed_interval fields were introduced in ES in 7.2, but I don't think Kibana will have support for it until 7.3 (I merged it a bit later and I don't think Kibana will have time to implement for 7.2).

Thanks for the detailed help btw! Definitely helps us identify issues easier/faster. :slight_smile:

7

Just as a quick update to this.
I was able to remidy the issue partially by adding the following in the JSON input for the X-Axis.
{"interval": "1440m"}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.