Allow Filebeat to add folder names as fields

(Ao) #1

One of the pieces of software we hoped to use Filebeat to collect logs for actually stores data in the file path to the log file. As executions are triggered via RESTful API, it builds a tree structure of folders based on each execution of the program. The folder names created hold a unique identifier that is not contained in the log files, but needs to be included on each event generated from that file.

Unfortunately, we've had to investigate alternatives to Filebeat as it cannot take information about the path of the log file and include it as a field in the event.

We would propose an enhancement whereby prospectors, which can already be defined with wildcard directory paths (ie. 'logs//.log'), are able to pass the directories that are matched to the wildcards back as fields in the event sent to the destination. Perhaps a special syntax, similar to regular expression capture groups. For example, the path could be defined as 'logs/%{*:1}/.log', and then the 'fields' configuration element could use %{1} to add the folder name matched by %{:1} into the event.

(Carlos PĂ©rez Aradros) #2

this is recurring topic lately, could you please open a feature request in github for it?


(ruflin) #3

Interesting idea. One part I'm missing: Each even from a file contains the file path. Couldn't you use that to extract this information in LS or ES Ingest?

(Ao) #4

You are absolutely correct, I had not considered this. A dissect or grok filter could be used to parse the source tag and obtain this information inside Logstash.

(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.