Configuration issues - Filebeat for shipping messages from a Kafka topic to Elasticsearch

Hi! I'd like to have Filebeat set up the way that it would consume messages from Kafka topic and then send them to Elasticsearch. Is there a way, how to set it all up in docker-compose? I could register Logstash as a Kafka consumer so I assume that it should be also possible for Filebeat. In the log, I can see that it loaded the config file. I don't see it, however, in the list of Kafka's consumers. So Filebeat doesn't consume any messages, even though they are in Kafka. Can somebody point me in the right direction? I assume that it should be a really basic configuration. Yet, I'm missing something.

The FIlebeat part of docker-compose.yml:
- kafka
container_name: filebeat
- "./filebeat.yml"

the content of the filebeat.yml file:
- type: kafka
- kafka:29092
topics: ["progress-raspberry"]
client_id: "filebeat"
group_id: "filebeat"

hosts: ["elasticsearch:9200"]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.