Kibana requesting too many doc values

Elastic Stack Kibana
1

We have an elasticsearch cluster that indexes some events that flow through our system for debugging use. These events have pretty widely varying formats, so they end up generating a lot of different fields. Kibana maps about 2000 fields for the indexes. We don't configure these indexes manually, and just let Elasticsearch automatically generate indexes based on the data.

Performance has never been a problem. Type conflicts are pretty rare and haven't been problematic enough to warrant any action.

Today, I refreshed field mappings, and all search queries are breaking with the following error. It looks like the indexes now have 101 different date type fields. Kibana seems to automatically request every date fields as docvalue fields in every single request.

These are for "Discover" requests, and we don't ever sort/aggregate on any of these fields. Is there a way to keep Kibana from requesting these fields as docvalue fields?

If not, how can we get Kibana working again? Update max_docvalue_fields_search for every index?

{"responses":[{"took":2480,"timed_out":false,"_shards":{"total":5695,"successful":5600,"skipped":5600,"failed":95,"failures":[{"shard":0,"index":"tracer--2018-09-07","node":"rMepPe8BS1m2ILlUDDQFmg","reason":{"type":"illegal_argument_exception","reason":"Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [101]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting."}}]},"hits":{"total":0,"max_score":0.0,"hits":[]},"status":200}]}

1 Like
2

Hi Yunchi, this is a tough one. I spoke to some other engineers on the team, and we think you have a few options to try:

  1. As you suggested, you could try increasing max_docvalue_fields_search for each index, though I have concerns this might not scale. If you continue to index documents with new fields then you'll probably bump up against this limit again.
  2. If you don't actually need to query/aggregate on all of those date fields, then maybe you could try mapping them as strings instead. Then they won't be requested as docvalue fields.
  3. You could also be trying to fit too many different types of data into a single index pattern, i.e. your index pattern is too greedy. Would your use case could allow you to define index patterns which match fewer indices, and thus include fewer date fields?

I hope this helps,
CJ

3

Hi CJ, thanks for asking the team. I ended up bumping max_docvalue_fields_search for now and that has worked out.

I think mapping them to strings is the next best option, and that's what I'll probably do when the next time this comes up. I really wanted to avoid having to create mappings manually though, since the schema is pretty arbitrary.

As for using a single index pattern, I do usually want my query to hit all indexes in Kibana. The events are usually for the same domain object, but within different stages of its life cycle, so querying by some common fields is good for reconstructing a sequence of events across systems. I believe that with a single query, I can only query one index pattern?

Why is there no way to limit which fields get queried as doc values in Kibana?

4

I believe that with a single query, I can only query one index pattern?

This is correct, but you can use commas in your index pattern to define one which matches different index naming schemes, e.g. foo*,bar* will match indices matching foo* as well as bar*.

Why is there no way to limit which fields get queried as doc values in Kibana?

I don't know the answer to this one, but it sounds like a good feature request. If you'd like to see this supported, could you please file an issue in GitHub with a summary of the functionality you'd like to see, a recap of why you need this, and a cross-link to this thread for reference?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

Just a note for anyone who might stumble across this old thread: The docvalues limit should no longer be causing problems starting in 7.11, as we shifted over to using the search fields API in this PR: https://github.com/elastic/kibana/pull/82383

1 Like