Include relative file to a log file, as a custom field

[PicoCreator (Eugene Cheah)]

Conceptual filebeat.yml configuration

filebeat:
  prospectors:
    paths: "/var/lib/docker/containers/*/*-json.log"
    relativeFields:
      container_name: "./hostname"

The idea is not to do a heavy parsing,
but create a utility to include dynamic field values,
relative to an individual log file. (./hostname is beside the log file)

These file can then be dynamically generated by other programs subsequently.
Or in this case docker itself [the main reason why i want this] =)

[ruflin]

You are probably looking for something like this? https://github.com/elastic/beats/issues/2186

An other option is to dynamically generate the configurations files and use fields for your additional info: https://github.com/elastic/beats/pull/3362

[PicoCreator (Eugene Cheah)]

Issue: Dynamic lookup processor

Looks close. Because in the docker example i given, it is possible to derive the hostname unique to each log file name, which can be passed in"Dynamic lookup processing scripts".

On the positive side, it is a lot more comprehensive and versatile solution. On the negative side, with the user permission safety, etc. It can be a lot harder to setup.

Implement prospector reloading

As for reloading of config, because each log file will have an individual field value unique to them. This will probably need a "FileBeatConfigBeat" to generate individual prospectors with field values for each log file.

[ruflin]

How is the hostname file generated? I assume this is already generated by some script, so this script could directly create the prospector config file.

[PicoCreator (Eugene Cheah)]

The hostname file is generated by docker itself in this case. So not much option in modifying that.

However since it is possible to scan for docker system structure, yup. Some bash script writing is probably needed, see how it goes. I may give it a try =)

Which version is prospector reloading in? and how do I trigger it?

[ruflin]

Prospector reloading is in master and the upcoming 5.3. The docs can be found here: https://www.elastic.co/guide/en/beats/filebeat/5.x/filebeat-configuration-reloading.html