How many Elasticsearch instances I should have in a node

[worapojc (Worapoj Chokeanankun)]

Hi Elasticsearch Team,

I'm using EC2 instance (i3.4xlarge) for hot-data node.

• Memory 122Gb
• Disk 2 x 1.9 NVMe SSD

There are 2 ES instances on this node. I assign the 2 disks for both instances.

I configure -Xmx31g -Xms31g for each instance.

From the picture, is it a good idea to add an instance with the same configuration?
The memory usage would be around 90+GB.

I also found GC overhead log. For example, [2019-05-10T18:34:51,377][INFO ][o.e.m.j.JvmGcMonitorService] [10.49.112.143-hdata-node-1][gc][5454] overhead, spent [367ms] collecting in the last [1.1s]

Will this increase the performance and resolve the GC overhead issue?

Thanks

[Christian_Dahlqvist]

It is recommended to use at most 50% of RAM for heap which is roughly what you have got at the moment. Elasticsearch required off-heap memory and relies on the OS page cache for performance so I would not recommend adding any further node.

Also, that log message does not necessarily indicate any GC problem. What does your heap usage look like over time? Do you have a nice saw-tooth pattern?

[worapojc (Worapoj Chokeanankun)]

Thanks @Christian_Dahlqvist.

I was configured 31G for 2 nodes in a host.

497 118705 1 99 May10 ? 15-14:47:27 /usr/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch/hdata-node-1 -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/10.49.112.143-hdata-node-1/hdata-node-1_elasticsearch.pid -d

497 121129 1 99 May10 ? 12-15:16:05 /usr/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch/hdata-node-0 -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/10.49.112.143-hdata-node-0/hdata-node-0_ elasticsearch.pid -d

I took the snapshot only first instance of this host

The JVM Heap looks like the nice saw-tooth pattern.

The indexing time...

At the right of Indexing Time (ms) graph, you will see that the value was growing rapidly because I added the third instance with the same configuration (heap size 31G)

I think the increasing of the indexing time relates to the shard relocation when I added the new node.

For the new node, the JVM still have a nice saw-tooth.

If Elasticsearch required off-heap memory, what will be the better option?

• 2 nodes with 31G heap size (50% of Memory)
• 3 nodes with 24G heap size (59% of Memory)

From JvmGcMonitorService log message, could you describe more details?