We have ELK deployment in which multiple logstash-forwarders (from multiple service logs) pushes log to logstash which then sends it to Kafka and then one more logstash pulls those logs from Kafka and indexes them in ElasticSearch cluster.
Can some one let me know:-
- What is the best way people handle traffic spikes in such deployment?
- Can we configure Kafka in such a way that a single logstash-forwarder (of a single service) does not hog the entire set-up? We want to ensure that all services can use the set-up fairly?
- How to ensure that one index does not grow huge compared to others? We have many daily indexes of several service logs and we want to ensure that during spike if one service index (or shards) become too large we block that service from sending logs, such that the cluster performance does not get effected?