I want to deploy ELK stack on Kubernetes for monitoring logs stored in Persistant Volume in the Kubernetes namespace. Logs are generating from different nodes, but are stored at the same place in Persistant Volume. What would be the best architecture to deploy ELK stack to monitor those logs.
Currently this is the architecture which I am using :
I have deployed Filebeat instance in a Pod in StatefulSet that collects logs from the PV and send it to Logstash. Only 1 Filebeat pod is running.
Logstash instance is also running in a Pod in StatefulSet and transfer these logs to Elasticsearch. Only 1 Logstash pod is running.
Elastic Search and Kibana is already running on some other cluster, so I don't have to worry about it.
I want to know if there is any other efficient way in which the Filebeat and Logstash instance can be deployed on Kubernetes. Is there any way in which 2 or more than 2 Filebeat instance could pick up logs from the same location and load balance among them?
I have already read about Load Balancing in Logstash, but want to achieve it at Filebeat level too.
Also, I want to make the Filebeat and Logstash Highly Available, so what are the best possible way to achieve it?