I cannot forward docker meta data on Kafka

[omerhanci (Omer)]

I am using elastic stack with Kafka which is sitting between logstash and filebeat. I used following configuration to add docker metadata to log.

    processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"

It was working before I added Kafka to the stack but now that metadata information is lost. Is there a way to forward that data over Kafka to logstash, then ES.

[shaunak (Shaunak Kashyap)]

Hi @omerhanci, welcome to the Elastic community forums!

A couple of questions/suggestions:

1. What version of Filebeat are you using?

2. Just for debugging purposes, could you temporarily set output.console.enabled: true and output.kafka.enabled: false and restart Filebeat? This will output events to the console log instead of sending them to Kafka. Then could you look at the events that are output and check if the Docker metadata fields are present?

Thanks,

Shaunak