Hi,
after upgrading to 7.12.0 a strange problem started occuring. Kibana keeps consuming more and more RAM until it goes to swap and then I have to restart kibana service because the performance goes wildly off. We have 3 kibana nodes at the moment, 2 of them have nginx proxy, but only one is for primary use (with UI), the second one with proxy is mainly for customers and the third one is there just to help with detection rules, so no UI use.
No matter the setup, even after commenting --max-old-space-size in node.options and /etc/default/kibana, the memory usage goes way over it should. Primary and third kibana are Ubuntu18, second kibana with proxy is Ubuntu16.
This screenshot is for second kibana with proxy (we still use internal monitoring and when node leaves the cluster, some kibana node disappears from monitoring, so I can't provide screenshot from primary kibana), where you can see, that --max-old-space-size have no real effect on memory usage:
but truth is, that memory heap is probably fine, so something else consumes the memory:
the same for primary kibana:
and screenshot for third kibana with no UI use:
Looking at htop, this is primary kibana now:
Secondary kibana:
Third kibana:
However, seeing this in Stack Monitoring is a bit funny:
So how could I troubleshoot what could be consuming memory so much? Could upgrading to 7.12.1 help? Didn't find anything in release notes, so don't know, what could be this issue related to.
Thanks.