this is a design investigation.
On many docker containers, we will have Filebeat that reads logs (generated by many services).
On one docker container we will have Elasticsearch that receives and indexes the logs.
We will have Kafka in one docker container for other purpose.
- Do we have advantages to let Kafka collect logs from the docker containers with Filebeat and forward them to the Elasticsearch container ?
- The Filebeat uses a backpressure-sensitive protocol when sending data to Elasticsearch. Won't we loose this advantage by putting Kafka in the middle ?
Thanks for your answers,