Another quick question around the same lines.
Would it make sense for filebeat to expose the filename that its reading from in order to determine the index name( or kafka topic for that matter).
For example,
if its currently tailing from a file "/var/log/docker/api.log" then,
include a field in the event{"event_source" : "api" } or even {"event_source" : "api.log" }along with other metadata?
I know that filebeat already exposes an absolute path in the source field, but that is not enough for determining which index / kafka topic to write to if you're doing this at scale. Also, adding a field in the event is going to require change in the way the events are logged which makes the transition to filebeat much more difficult.
What we need is something very similar to : Using filename from filebeat in index pattern