x-pack will report around 50% heap use, and based on the graphs in x-pack it seems to do a full gc around every hour, still I am experiencing performance problems and seeing:
The new generation is surprisingly small, and that is what is driving all the ParNew activity. Can you remove these? I am wondering if that is influencing the choice of NewGen size even though they are overriden later.
That's OK, that is not the issue. Something is deliberately setting MaxNewSize low. With a 6 GB heap then I see this in my gc.log ... -XX:MaxNewSize=174485504. That seems crazy low to me but it wasn't put there by the JVM option fairy, so there must be a reason.
If I add -XX:MaxNewSize=2147483648 to my jvm.options then it is respected. I still get terrible performance from ParNew, just a lot less frequently.
Personally I run logstash using the G1 garbage collector, but I believe that is unsupported by Elastic.
This is elastic, not logstash. I have looked more into the figures, and it seems the young generation is running very often, but there is a lot of free heap, and the old generation runs only about twice an hour.
I tried removing the -Xmx1g and -Xms1g that's there by default in the settings, but it did not influence it in any way. I think the defaults set are not sane.
In logs/gc.log the third line should have the command line flags. If it does not you can enable it using -XX:+PrintCommandLineFlags. What is the value of -XX:MaxNewSize= ?
To me telling the JVM to use 166 MB out of an 18 GB heap for the new generation is crazy. I wish I could find where that gets set, or even better, why someone thought it a good idea to set it. They must have had a reason.
What do things look like if you set -XX:MaxNewSize=2147483648 in your jvm.options?
checking the Elasticsearch log about the heap size when it starts. For example, below I've started with "ES_JAVA_OPTS=-Xms3G -Xmx3G": [2018-04-12T07:39:06,417][INFO ][o.e.e.NodeEnvironment ] [4b5QpNp] heap size [2.9gb], compressed ordinary object pointers [true]
Yes the actual heapsize can be set with the parameters, but what is not so good are the sizings of the young gen, see comments from Badger above on "MaxNewSize"
Just for reference for future readers that might find this thread: The JVM determines the size of the young generation ergonomically (i.e. based on the capabilities of the hardware). One of the parameters that influence this decision is the number of available GC threads which is in turn based on the number of available CPUs on that machine. On said machine the JVM had only two GC threads which lead to such a small young generation size. The full analysis with more details is available on Github.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.