We are currently in the process of building out a new Elasticsearch cluster, with server nodes of 64GB of memory, of which 62GB are dedicated to Elasticsearch process. My question is, we noticed that by default, the java process is set to "-XX:+HeapDumpOnOutOfMemoryError". I assume that this would mean that if an out-of-memory error should occur, it will dump a 31G memory dump file do the default location? If that is the case, is it recommended to leave it on?
Yes.
It depends, do you want to be able to debug an OutOfMemoryError? Note that you can add "-XX:HeapDumpPath=/path/to/heapdump/heapdump.hprof" to JAVA_OPTS to specify the location.
If we want to just disable this, what would be the correct way to?
Comment out
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
in elasticsearch.in.sh. I advise very strongly against this. If you're running into situations where having a heap dump is a problem, then all you're doing is preventing the production of useful information that can be used to debug why you're running into OutOfMemoryErrors. If you're not running into situations where having a heap dump is a problem, then all you're doing, should the need ever arise, is preventing the production of useful information that can be used to debug an unexpected OutOfMemoryError.
That is, the solution to the problem of heap dumps filling up disk space is not to turn them off, it's to figure out why the heap is dumping in the first place and the heap dumps will help you figure that out.
So, returning to this question, the answer is yes.
will the JAVA_OPTS line remain commented even after i update logstash?
or i will have to comment it again, after every update?
It's not easy to give a definitive answer to this question because there are many varied ways of installing and they don't all do exactly the same thing. The simplest way to get a definitive answer to your question for your setup is to just try it out.
But again, I strongly advise against disabling this flag. If you're having an issue with heap dumps filling up disk space, the root problem is not that your disk is being filled with heap dumps, that is just a symptom of the real problems. And heap dumps are one thing that you need to actually address one of the real problems.