One of the pieces of software we hoped to use Filebeat to collect logs for actually stores data in the file path to the log file. As executions are triggered via RESTful API, it builds a tree structure of folders based on each execution of the program. The folder names created hold a unique identifier that is not contained in the log files, but needs to be included on each event generated from that file.
Unfortunately, we've had to investigate alternatives to Filebeat as it cannot take information about the path of the log file and include it as a field in the event.
We would propose an enhancement whereby prospectors, which can already be defined with wildcard directory paths (ie. 'logs//.log'), are able to pass the directories that are matched to the wildcards back as fields in the event sent to the destination. Perhaps a special syntax, similar to regular expression capture groups. For example, the path could be defined as 'logs/%{*:1}/.log', and then the 'fields' configuration element could use %{1} to add the folder name matched by %{:1} into the event.