Can't find terms that are not ".keyword" and the "something.keyword" terms are empty

Hi, I've been playing a little bit with mappings and index templates lately, and that actually helped me make the SIEM work and see a lot of improvement. However, many of my visualizations just stopped working, mainly because they used to depend on "foo.keyword" terms that now are apparently empty... In many cases I could find the corresponding term without the ".keyword", but in some others I can only find "food.keyword" and therefore can't fix the visualization. My guess is that it has something to do with mapping, but I have no clue how to fix things. I've tried reindexing, refreshing index patterns, assigning legacy templates, but nothing. Could you please help me?

If you change the underlying mapping, visualizations might stop working because they are referencing the old field names. The only way to fix it is going through all visualizations and fixing the problems by hand.

If a field went away, then you might have renamed it or don't index it as keyword field type anymore (this is necessary for visualizations because text fields are not aggregatable).

If you are stuck at specific places, feel free to post the context (your visualization and the mapping of your indices) and I can help you fixing them.

1 Like

Thank you, I think I fixed it but I'm not entirely sure how. I did 3 things:

  1. Created an index template with very few mappings for source and host objects, and assigned it to my index templates.
  2. Copied a Legacy Index Template from logstash-ecs and assigned it to my index templates as well.
  3. Created a new index template that uses comma-separated indices rather than a wildcard.

I am guessing the one that actually did the trick was 1 and 3. But after doing that and reindexing, I was able to aggregate the terms again. Also played definingmessage as keyword instead of text which made it available for aggregation once again, but I now understand why is not recommended.