I'll start off with a brief summary of the setup:
Single node, ES 6.4.2, Debian 9 with OpenJDK 1.8.0_212
+/- 500 indexes, 2500 shards, 12M documents, 3.5 GB data.
x-pack monitoring, 60s samples, 7 days retention
What we are experiencing is that the JVM heap is slowly creeping up. Once it hits 100%, latency, segment count and GC duration go up, where GC Young drops to 0. Queries take very long causing the website that uses the ES to become unresponsive/unusable. Restarting Elasticsearch solves the problem.
Is there any way we can figure out what's happening here or what is filling up the heap, and how we can prevent this from happening?