We are running logstash on Kubernetes. And some logstash Pods stops inserting data into ES for some unknown reason periodically (once in 2-7 days)
And logstash does not indicate there's something wrong with it. I mean the logstash process does not crash.The logs do not show any errors.
The only way I may know that the logstash stopped inserting data into ES is by looking into kibana and see no new data.
And restarting logstash is fixing the issue or after 2-4hrs it starts working normal again.
Has anyone had that kind of issues before? How can I identify whats causing this issue and fix it?
Logstash is receiving data from Filebeat for every
30sec interval. It is receiving data from
~70 filebeat pods and load-balancing enabled on filebeat logstash output section to distribute load across all logstash pods
Logstash version - 7.6.2
LS_JAVA_OPTS=" -Xmx4g -Xms4g"
Number of logstash pods - 8