Empty drop down list: Filtering occurs on field which does not exist

Elastic Stack Kibana
1

Hi there, we're having problems using a dashboard containing a drop down list control. The drop down wont show options, even though direct search for the given keyword does return results, i.e. those keyword do exist (however, not on all documents in the index pattern).

2019-07-17-085938_2707x2113_scrot|690x402

Analyzing the queries sent by Kibana, the query for the drop down does not contain a time range. When adding a time range to that query, the correct results are returned. Thanks for any pointers!!

Original query without time range, which does not return any hits (this is the type of query sent by Kibana for a drop down control):

{
  "_source": {
    "excludes": []
  },
  "aggs": {
    "termsAgg": {
      "terms": {
        "size": 10,
        "order": {
          "_count": "desc"
        },
        "field": "kubernetes.labels.applicationType.keyword"
      }
    }
  },
  "docvalue_fields": [
    "@timestamp",
    "metadata.creationTimestamp",
    "requestObject.metadata.creationTimestamp",
    "requestObject.status.conditions.lastProbeTime",
    "requestObject.status.conditions.lastTransitionTime",
    "requestReceivedTimestamp",
    "responseObject.metadata.creationTimestamp",
    "responseObject.metadata.deletionTimestamp",
    "responseObject.spec.taints.timeAdded",
    "responseObject.status.conditions.lastHeartbeatTime",
    "responseObject.status.conditions.lastProbeTime",
    "responseObject.status.conditions.lastTransitionTime",
    "responseObject.status.containerStatuses.lastState.terminated.finishedAt",
    "responseObject.status.containerStatuses.lastState.terminated.startedAt",
    "responseObject.status.containerStatuses.state.running.startedAt",
    "responseObject.status.startTime",
    "stageTimestamp",
    "time",
    "timestamp"
  ],
  "query": {
    "bool": {
      "must": [],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "script_fields": {},
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "terminate_after": 100000,
  "timeout": "1s"
}

The modified query including a time range, which does return the desired keywords (this is the type of query sent by Kibana for a tag cloud):

{
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "terms": {
        "field": "kubernetes.labels.applicationType.keyword",
        "size": 5,
        "order": {
          "_count": "desc"
        }
      }
    }
  },
  "docvalue_fields": [
    "@timestamp",
    "metadata.creationTimestamp",
    "requestObject.metadata.creationTimestamp",
    "requestObject.status.conditions.lastProbeTime",
    "requestObject.status.conditions.lastTransitionTime",
    "requestReceivedTimestamp",
    "responseObject.metadata.creationTimestamp",
    "responseObject.metadata.deletionTimestamp",
    "responseObject.spec.taints.timeAdded",
    "responseObject.status.conditions.lastHeartbeatTime",
    "responseObject.status.conditions.lastProbeTime",
    "responseObject.status.conditions.lastTransitionTime",
    "responseObject.status.containerStatuses.lastState.terminated.finishedAt",
    "responseObject.status.containerStatuses.lastState.terminated.startedAt",
    "responseObject.status.containerStatuses.state.running.startedAt",
    "responseObject.status.startTime",
    "stageTimestamp",
    "time",
    "timestamp"
  ],
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  },
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1563259133739,
              "lte": 1563345533740,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "script_fields": {},
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "version": true
}
2

Alright, the problem is as follows:

...
"terminate_after": 100000
...

limits the number of documents considered per shard. In our case, these documents did not include the newest documents, and only those newest documents have the desired keywords.

We did not find an option to change the value for terminate_after. :frowning:

However, when activating the 'Respect time range' option on the dropdown control, the query now considers relevant documents. Thus the dropdown lists are suddenly filled with the desired keywords. :slight_smile: