Field added by Logstash Grok can't be used in Kibana filters

Elastic Stack Kibana
1

I added a hostname field to my index using Logstash Grok:

grok {
           match => { "originsicname" => "CN=%{HOSTNAME:hostname}," }
...

For some reason I cannot use this hostname field as a filter on my dashboards. The Kibana dropdown control panel says this field doesn't exist on any documents in the index pattern. And the dashboard Edit Filter dialog says There aren't any options available. when I try to create a filter.

Screen Shot 2021-03-12 at 11.57.04 AM
Screen Shot 2021-03-12 at 11.57.04 AM898×734 68.6 KB

Yet the Discover app shows the field does exist in the index.

discover-hostnames
discover-hostnames1124×602 36.3 KB

And the Management app shows my Index Pattern has the field and is searchable and Aggregatable:

index-pattern-field
index-pattern-field2334×840 61.5 KB

On another dashboard with a separate (but nearly identical) index pattern, I'm able to use a hostname field as a filter in Kibana just fine. The only difference is on the other index, I didn't have to add the hostname field using Grok. I'm using the filebeat-7.11.0 index pattern as a component template for both index patterns. Filebeat is the original input source to Logstash for both indices. I'm running ES 7.11 on Debian.

Any advice?

2

Does the index pattern one that works have a hostname.keyword mapping while the one that does not is missing that?

3

Both of my index patterns just use the same component template filebeat-7.11.0. I see the same inherited mappings on both indices:

GET mcs-checkpoint-2021.03.05-01/_mapping
{
  "mcs-checkpoint-2021.03.05-01" : {
    "mappings" : {
...
        "checkpoint" : {
...
            "hostname" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
...

I just tried creating a checkpoint.hostname field with logstash grok to see if I was just not using the correct namespace, but that seems to have the same issue.

4

Ah, you were right afterall. I've just rebuilt the index mapping just as before, but this time specified explicitly the mapping:

{
  "properties": {
    "hostname": {
      "type": "keyword"
    }
  }
}

And now Kibana is able to filter using this keyword as expected.

5

That's good because I was stuck on what to check next. :slight_smile: