We're using a lot filebeat to scrape log files and send them to our ELK infrastructure. We're sending logs from many systems, and want them isolated by their own indices.
With filebeat, it's easy: we use the fields config at the prospector level to help us do that.
filebeat: prospectors: - paths: - /path/to/log/files/*.log [...] fields: elk: client: <client code> index: <target index> ... fields_under_root: true
So, I thought that it could be the same method used for all the beats....
Looking to packetbeat, it doesn't seems...
Is there a way to force custom fields to all beats?
Can it be easily added?
Also, in filebeat it can be done at a specific prospector level. In our case it's perfect. But would be nice to be able to specify fields at all levels of config:
- beat, default for all events spooled out from the beat
- filebeat - prospector level: be able to be more specific by prospector level
- packetbeat - protocol level: be able to be more specific by defined protocol
- ... and so on ...
If it's not possible, we must create a dedicated input (and opening ports) per originating clients/system to assign them theses values at input level...
Don't know if it's bad because each input generally creates a thread.... ?