Strange Logstash index fields in visualization

Elastic Stack Kibana
1

Hello, all.

I previously posted this in the Logstash forum, but was advised that this may be a better location. I hope I present my issue understandably.

I'm running Elastic Stack 5.6, and most things are working well. However, when I try to create a Logstash-based visualization in Kibana, and use "Terms" or "Significant Terms" for aggregation, I'm presented with fields in the pick list, which are strange to me. They look like this:

AccountType.keyword
Action.keyword
ActivityID.keyword

So, ".keyword" seems to be appended to all of the fields. I see the same fields in the "Index Patterns" in Kibana, and they list as "Aggregatable". But, they don't seem to be.

What am I doing wrong?

Thanks.

2

the .keyword fields are multi-fields that are created for your fields mapped as text. When fields are mapped as text, they are tokenized by the analyzer so what is actually stored in Elasticsearch is a collection of tokens. So, although you could aggregate on that field, the result probably wouldn't be what you are expecting.

The .keyword field is a non-tokenized copy of the text, and therefore should be aggregatable.

When you say:

I see the same fields in the "Index Patterns" in Kibana, and they list as "Aggregatable". But, they don't seem to be.

What do you mean?

3

Thanks for the reply, BigFunger.

Well, OK, never mind about this: "I see the same fields in the "Index Patterns" in Kibana, and they list as "Aggregatable". But, they don't seem to be." But, are the .keyword fields the "usual" ones I should see, or should they be something else. I see the .keyword fields when I try to do a visualization when I try to use the logstash index. In fact, most of the fields available are of the .keyword variety. Would it be helpful for me to post my config files?

4

The .keyword fields are added because text fields are not really aggregatible in a meaningful way. If you have fields in your documents that you really want to be able to aggregate on the verbatim values, make sure you define them in your mapping as keyword type instead of text. If you are relying on the mappings created automatically by Elasticsearch (which in production, you shouldn't be) then string values are automatically mapped as a text field with a keyword multi-field.

ie:

# NOTE: text-test index does not exist.
POST text-test/doc/1
{
  "name": "jim unger"
}

GET text-test/_mapping

returns this:

{
  "text-test": {
    "mappings": {
      "doc": {
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

Here is a blog post with more information:

5

Thanks for the explanation. Alas, I'm not good at what I'll call "Elastic programming", so I don't know how to create/define mappings. I did run "GET filebeat-2017.09.20/_mapping", and have posted it here ( https://pastebin.com/RrgDSNUb ), in case you're kind enough to take a look and, maybe, show me how I can change the mapping, if necessary.

Again, thanks.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.