As suggested in the ECS documents I want to set the
ecs.version field in my Logstash pipelines. But since I'm using multiplie cascading pipelines (and receive events via filebeat), sometimes the field is already set and I end up with multiple values in one field.
What should I do? Clean up the field before setting? Leave all values?
To be more clear about what I'm doing:
- I receive messages via filebeat (which seems to set
- I process the messages through a
syslogpipeline which parses the syslog header using an out of the box
grokpattern which doesn't honor ECS
- Only the log events containing
postfixin programm are afterwards processed by a
postfixpipeline which sets the
So I end up with an event that:
- has some fields correctly set according to ECS 1.4.0 and the value
- has some fields that don't fit into ECS at all (like
pid) but still
- has some fields conforming to ECS 1.5.0 due to the potfix pipeline
What should I do?
- Always check for the existence of the field and in case remove and set it to the highest version?
- Have an extra pipeline to rename the fields and set the version? (seems hard to handle and like wasting lots of resource)
- Set only the minimum version all of the event is conforming to
- what else?