We are working on designing ELK cluster for our client. We came up with the below design approach.
- Filebeat will source the data from log files, add tags and send it to Kafka Producer
- Kafka consumer will read from Kafka producer and source it to the logstash
- Logstash will parse the logs based on the tags and index to the Elastic Cluster.
We have got questions related to the best design approach
- We have different clients, is it good to have individual filebeat configuration files for each client or can we have the configuration in one filebeat configuration file.
- Similarly is it good to have individual logstash file configuration for each clients.
- To load balance the writing speed of logstash into Elastic cluster do we need a load balancer like Kafka in between them