Hi LS community,
Are you aware of a filter that can compare the list of fields in an event against those in a ES mapping template (JSON file)? Ideally, the filter would then "isolate" those fields that are found in the event, but not the mapping. The rogue fields could all be placed into a inner object, whose name is controlled through the filter configuration. As an example, all unknown fields could be put under an object called: "unrecognised_fields".
The reasons I would like a filter like this are to:
- Allow us to use a strict mapping for all fields except for the particular inner object that contains all the unknown fields.
- Help us with our automated testing of logstash configuration (finding situations where we haven't properly updated our patterns or filter configuration). Would be much easier than looking for mapping parser exceptions from ES.
If I were to write such a filter, do you think it would be useful to other LS users?
Thanks!
Nick George