Kibana using docvalue_fields for no apparent reason

[Michiel_Van_Der_Lee]

I have a simple visualization:
Metrics: Count
Buckets:

• Split group: Date Histogram, @timestamp, Weekly

Kibana builds the following request:

{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "1w",
        "time_zone": "America/Detroit",
        "min_doc_count": 1
      }
    }
  },
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2020-06-09T12:20:30.505Z",
              "lte": "2020-06-09T12:35:30.505Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    ...
  ]
}

My question. Why are there 100+ "docvalue_fields" that are causing errors when I'm only requesting a count grouped by a single column? And do I have any control over it?

This seems incredibly retarded to me!

[flash1293 (Joe Reuter)]

Kibana automatically requests doc values for all time fields in the index pattern instead of working with the _source object directly to avoid running into problems parsing different date formats.

I agree that these doc values are not required for pure aggregations, but they shouldn't do any harm as well because the query doesn't fetch any hits.

What errors are popping up for you? If one of those fields is not part of your index anymore, it should be enough to refresh the index pattern in Management > Index patterns . This will sync the internal index pattern representation within Kibana with the actual data indices in your cluster.

[Michiel_Van_Der_Lee]

Refreshing the index pattern is what broke everything for us.

{"type":"illegal_argument_exception","reason":"Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [132]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting."}},

Yeah I can update it for all our indices.. But why do I have to? I'm not requesting that many fields, I have no need for it.

Managing multiple indices for multiple clusters this is tedious task and an unnecessary waste of my time.

[flash1293 (Joe Reuter)]

It seems like you have more than 100 date fields in your index pattern. As mentioned, these are only necessary for the Discover view, but will end up in all requests.

It definitely makes sense to create a bug report for this in the Kibana repo: https://github.com/elastic/kibana/issues/new?template=Bug_report.md

But I'm not sure whether just omitting the doc values for visualization queries is a good approach because half of Kibana will still be broken (mainly Discover and saved searches on dashboards). At the moment there is no way around bumping the limits for your indices. Index templates can help making this easier from now on.

This is only necessary for index patterns that cross the 100 date fields threshold.

[Michiel_Van_Der_Lee]

Okay, thanks Joe