Kibana filter window

[Tuckson (Eric)]

Hi,

In kibana (currently 7.14.1) we often use the filter window when working on the 'discover' pages.

One chooses the concerning field, the operator and then in the past in the past the value field had a dropdown which had some values which where better matched when you started typing. However now it's just empty most of the times.

Any idea why? How is it populated anyway?

All our own indices are datasream, but the same problem arises with (for example) the metricbeat indice.

Thnx for your thoughts

[bhavyarm]

Hello,

Can you click plus on the field under documents table and see if the filter works? and you see a filter added on top and can edit on it?

Thanks,
Bhavya

[Tuckson (Eric)]

Hi, yes, that works.
I can also manually enter an existing value and that works as well. It's just that the field in the filter window is not automatically populated.

[bhavyarm]

Is the index pattern on which you are querying is an alias by any chance?

Also what do you get for field caps?

Field capabilities API | Elasticsearch Guide [7.14] | Elastic

Thanks,
Bhavya

[Tuckson (Eric)]

Yes, alias as far as I know. Most patterns point to datastreams.

Will look into those caps later today ( it's midnight here now )

[Tuckson (Eric)]

Hi,

When doing this field caps thing I get a huge list of indices in return, including the one of the datastream Iam looking to most of the time.
look like this:

#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
#! this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .kibana_1, .kibana_2, .kibana_3, .kibana_4, .kibana_5, .kibana_6, .kibana_7, .kibana_7.12.0_001, .kibana_7.13.0_001, .kibana_7.13.2_001, .kibana_7.14.0_001, .kibana_7.14.1_001, .kibana_task_manager_1, .kibana_task_manager_2, .kibana_task_manager_3, .kibana_task_manager_7.12.0_001, .kibana_task_manager_7.13.0_001, .kibana_task_manager_7.13.2_001, .kibana_task_manager_7.14.0_001, .kibana_task_manager_7.14.1_001, .reporting-2020-11-29, .reporting-2021-02-14, .reporting-2021-03-07, .reporting-2021-06-13, .reporting-2021-06-20, .reporting-2021-07-11, .tasks], but in a future major version, direct access to system indices will be prevented by default
{
  "indices" : [
    ".apm-agent-configuration",
    ".apm-custom-link",
    ".async-search",
    ".ds-agl-api-ds-2021.08.10-001404",
    ".ds-agl-api-ds-2021.08.10-001405",
    ".ds-agl-api-ds-2021.08.10-001406",
    ".ds-agl-api-ds-2021.08.11-001407",
    ".ds-agl-api-ds-2021.08.11-001408",
    ".ds-agl-api-ds-2021.08.11-001409",
    ".ds-agl-api-ds-2021.08.12-001410",
    ".ds-agl-api-ds-2021.08.12-001411",
    ".ds-agl-api-ds-2021.08.12-001412",
    ".ds-agl-api-ds-2021.08.13-001413",
< another long list of indices >
  ],
  "fields" : { }
}

[bhavyarm]

Eric,

Are you logged in as super user or a different user with adequate privileges?

Do you think you are seeing this issue? https://github.com/elastic/kibana/issues/111642 I logged this bug for a different user yesterday.

Thanks,
Bhavya

[bhavyarm]

Eric,

So we tracked this down and I have two bugs logged - https://github.com/elastic/elasticsearch/issues/77508 and https://github.com/elastic/kibana/issues/111642

The workaround is you change autocomplete:valueSuggestionMethod to terms_agg in advanced settings in Kibana

Thank you,
Bhavya

[Tuckson (Eric)]

Thnx. Will try the work around tomorrow
as for your other question, I don't have security enabled on this cluster, so you tell me which role I have. Definately not superuser since kibana for certain features tells me I need to be one

Thnx again Bhavya, you're the best.

Regards, Eric

[bhavyarm]

@lukas Eric mentioned that they don't have security enabled.

Thanks,
Bhavya

[lukas (Lukas Olson)]

As far as how this list is populated, by default in 7.14+ we use the performant Elasticsearch _terms_enum API. Here's an example of the query we send:

POST my_index_name/_terms_enum
{
  "field": "ms-result",
  "string": "",
  "index_filter": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": "2021-09-09T20:40:00.000Z",
            "lte": "2021-09-09T20:55:59.999Z",
            "format": "strict_date_optional_time"
          }
        }
      },
      {
        "terms": {
          "_tier": [
            "data_hot",
            "data_warm",
            "data_content",
            "data_cold"
          ]
        }
      }
    ]
  }
}
}

The range filter corresponds with the selected time range, and the "string" section corresponds with what you've typed already in the filter bar.

(If you go into advanced settings and select "terms_agg" for "value suggestion method", then we revert to pre-7.14 behavior of using a terms aggregation for these value suggestions.)

A couple of questions: Does switching to terms_agg fix your problem? Does the value you're trying to filter for exist in the currently selected time range? What is your selected time range? (We have a bug related to this.)

[Tuckson (Eric)]

Hi,

Thnx.
Yes, going back to terms_agg fixed the problem.
The value I am trying to filter is not relevant. Pre 7.14. if you haven't typed a single character but instead clicked om the dropdown it already expands with a number of suggestions, which are changed upon further typing. the default time range I tested with was 15 minutes.
I do however recognize that there are situations we DID get some suggestions. Unfortunately I did not see way yet to reproduce that.

I am testing a bit further, and what even if I start typing a value which 100% sure is inside the timerange, it still is not suggested.

Did some testing,
When I use the index directly, I get results.

POST .ds-agl-api-ds-2021.09.09-001422/_terms_enum
{
  "field": "ms-result",
  "string": "",
  "index_filter": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": "2021-09-09T20:40:00.000Z",
            "lte": "2021-09-10T20:55:59.999Z",
            "format": "strict_date_optional_time"
          }
        }
      },
      {
        "terms": {
          "_tier": [
            "data_hot",
            "data_warm",
            "data_content",
            "data_cold"
          ]
        }
      }
    ]
  }
}
}

gives:

{
  "_shards" : {
    "total" : 8,
    "successful" : 8,
    "failed" : 0
  },
  "terms" : [
    "0",
    "1",
    "10",
    "100",
    "101",
    "1010",
    "102",
    "1020",
    "1024",
    "103"
  ],
  "complete" : true
}

When I replace the indexname with the datastream name I still get results in devtools

{
  "_shards" : {
    "total" : 712,
    "successful" : 712,
    "failed" : 0
  },
  "terms" : [
    "0",
    "1",
    "10",
    "100",
    "1002",
    "101",
    "1010",
    "1014",
    "1015",
    "1016"
  ],
  "complete" : true
}

[Tuckson (Eric)]

So it looks like the issue is related to the discover page filter window and not the query itselves? When I type the filter into the search bar, there are results:

However are you aware that both methods at first give a different resultset?

(problably sorted vs unsorted?)