Don't send tags to ElasticSearch

Elastic Stack Logstash
1

I am processing kafka events and sending them to ES. I tag events so that I can tell which kafka topic they were from. I then use the tag in the output to direct events from different kafka topics to different ES indexes.

However, beyond their use in Logstash, I do not wish to send the tags to ElasticSearch and have to see them in Kibana. Is there any way to use the tags in logstash but prevent them from being sent to ES?

2

You can have one final filter that removes the tags. There's no way to filter fields in the events in the elasticsearch output.

3

Two options of the top of my head...

  1. Use @metadata.tags instead of tags. @metadata fields are not forwarded by outputs, including the elasticsearch output. Your pipelines will need to use add_field instead of add_tag to populate this field. Or you can use add_tag and as the last step before your outputs rename tags to @metadata.tags. This will allow you to use @metadata.tags in your outputs to specify index names, but the field itself won't be sent.

  2. Use an elasticsearch ingest pipeline to strip the tags field out of the incoming data before it is indexed.

4

@rcowart Using @metadata worked. Thank you both very much for the suggestions.

Regarding "elasticsearch ingest pipeline to strip the tags field out of the incoming data before it is indexed." are you referring to making a mapping or a dynamic mapping template for that field in that index such that "index": false? Would that be equivalent to stripping that field from the incoming data? Or does it mean that the field is still there but not "indexed" (meaning I can't search by it, but it is still there if I search by other fields?)?

Thank you.

5

I mean using the ingest node functionality of Elasticsearch. You can read more here...

https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.