We are running logstash on Kubernetes. And some logstash Pods stops inserting data into ES for some unknown reason periodically (once in 2-7 days)
And logstash does not indicate there's something wrong with it. I mean the logstash process does not crash.The logs do not show any errors.
The only way I may know that the logstash stopped inserting data into ES is by looking into kibana and see no new data.
And restarting logstash is fixing the issue or after 2-4hrs it starts working normal again.
Has anyone had that kind of issues before? How can I identify whats causing this issue and fix it?
Logstash is receiving data from Filebeat for every 30sec interval. It is receiving data from ~70 filebeat pods and load-balancing enabled on filebeat logstash output section to distribute load across all logstash pods
Logstash version - 7.6.2 pipeline.workers: 12 LS_JAVA_OPTS=" -Xmx4g -Xms4g" Number of logstash pods - 8