Custom fields posibility

Elastic Stack Beats
1

Hello,

We're using a lot filebeat to scrape log files and send them to our ELK infrastructure. We're sending logs from many systems, and want them isolated by their own indices.

With filebeat, it's easy: we use the fields config at the prospector level to help us do that.

Example:

filebeat:
  prospectors:
    -
      paths:
        - /path/to/log/files/*.log

      [...]
      
      fields:
        elk: 
          client: <client code>
          index:  <target index>
          ...

      fields_under_root: true

So, I thought that it could be the same method used for all the beats....
Looking to packetbeat, it doesn't seems... :frowning:

Is there a way to force custom fields to all beats?
Can it be easily added?

Also, in filebeat it can be done at a specific prospector level. In our case it's perfect. But would be nice to be able to specify fields at all levels of config:

  • beat, default for all events spooled out from the beat
  • filebeat - prospector level: be able to be more specific by prospector level
  • packetbeat - protocol level: be able to be more specific by defined protocol
  • ... and so on ...

If it's not possible, we must create a dedicated input (and opening ports) per originating clients/system to assign them theses values at input level... :dizzy_face:

Don't know if it's bad because each input generally creates a thread.... ?

Thanks
Bruno Lavoie

2

In version 5 we added the ability to set fields at the beat level. It's all pretty well explained in this pull request. I didn't add it to each protocol in Packetbeat, but it wouldn't be too hard to do if you were interested in contributing.

Docs: https://www.elastic.co/guide/en/beats/filebeat/master/configuration-shipper.html#libbeat-configuration-fields

3

Nice, as our upgrading schedule is sometime slow, is it easy/possible to backport it to a current stable release?
Thanks
Bruno Lavoie

4

Anything is possible :smirk:, but it probably would be a bit a work since this feature touched a lot files (see PR). Also I don't think there is another point release planned for 1.x at the moment, only releases to fix bugs.