Hi! I'd like to have Filebeat set up the way that it would consume messages from Kafka topic and then send them to Elasticsearch. Is there a way, how to set it all up in docker-compose? I could register Logstash as a Kafka consumer so I assume that it should be also possible for Filebeat. In the log, I can see that it loaded the config file. I don't see it, however, in the list of Kafka's consumers. So Filebeat doesn't consume any messages, even though they are in Kafka. Can somebody point me in the right direction? I assume that it should be a really basic configuration. Yet, I'm missing something.
The FIlebeat part of docker-compose.yml:
filebeat:
depends_on:
- kafka
image: docker.elastic.co/beats/filebeat:7.9.2
container_name: filebeat
volumes:
- "./filebeat.yml"
the content of the filebeat.yml file:
filebeat.inputs:
- type: kafka
hosts:
- kafka:29092
topics: ["progress-raspberry"]
client_id: "filebeat"
group_id: "filebeat"
output.elasticsearch:
hosts: ["elasticsearch:9200"]