Resolved - [ES7.0]Terms aggregration doesn;t aggregrate by individual values


cheriemilk

2

2m

Hi team,
I am running into an issue in ELK production environment and can’t figure out why googled. The google result is saying that it might because the ktf1.raw field not enabled. But I still failed after apply the raw field. Can you help have a look and advice?

Issue Description:

  1. I have below data in elk, the value for ktf1, kf1, kf2, ktf3 are arrays. In Kibana, I want the it aggregate by the values in the array. But I didn’t get the expected result. It treat the array as a string, and aggregrate based on the string. Please refer to the screenshot in point3 .

    at,ktf1,kf1,kf2,kf3
    SAVE,"[performance,compensation]","[true,true,false,false]","[false,false,false,false]","[false,false,false,false]"
    SAVE,"[performance,compensation,Others]","[true,true,true,false]","[false,false,false,false]","[true,true,false,false]"
    SAVE,"[performance,compensation,Others]","[true,true,true,false]","[false,false,false,false]","[true,true,false,false]"
    SAVE,"[performance,compensation,Others]","[true,true,true,false]","[false,false,false,false]","[true,true,false,false]"
    SAVE,"[performance,compensation,liveProfile,talentFlag,Others]","[true,false,false,false]","[false,false,false,false]","[true,false,false,false]"

  2. The index template definition for ktf1, kf1, kf2, ktf3 are below.

    "template": "kvaudit",
    "index_patterns": ["kvaudit"],
    "settings": {
    "index": {
    "number_of_shards": "1",
    "codec": "best_compression",
    "number_of_replicas": "0"
    }
    },
    "mappings": {
    "doc": {
    "properties": {
    "@version": {
    "type": "keyword" }
    }
    }
    },
    "beat": {
    "properties": {
    "version": {
    "type": "keyword"
    }
    }
    },

    "fields": {
    "properties": { "at": {"type": "keyword"},
    "ktf1": {"type": "text","fields": {"raw": {"type": "keyword","ignore_above": 256}}},
    "ktf2": {"type": "keyword"},
    "ktf3": {"type": "keyword"},
    "ktf4": {"type": "keyword"},
    "ktf5": {"type": "keyword"},
    "kf1": {"type": "keyword"},
    "kf2": {"type":"keyword"},
    "kf3": {"type":"keyword"}}}

  3. Screenshot of Terms aggregration

Instead of indexing a string with an array of values like this:

"[performance,compensation]"

You need to index an array of string values like this:

["performance","compensation"]

1 Like