I suspect the answer to this is that "filebeat can't do that", but figured I'd ask anyway in case I'm missing some feature.
What I'm trying to do is harvest docker container logs and redirect them to different files based off of some attribute.
For example, if I got the following log line:
I would like to direct this to an output file named "jobid-11.log", based on the jobid in the above json.
The problems I see with this are:
- Filebeat doesn't seem to be able to parse fields into any sort of variable; It's simply got limited parsing ability for purposes of filtering.
- Filebeat doesn't seem to be have any concept of variables that can be used to change the output file name.
Even if #1 is true, I could still make this work if I could take something like the "source" output field and redirect to a different file based off of some substring of that (container id), but that would still require #2 to be true.
Is filebeat simply the wrong match for what I'm doing? I'm sure I could get logstash to do this, but I was looking for a lighter weight solution.