We have a 9 box setup (details provided in the Environment section further down).
We have a job that uses the bulk write interface to write 100 documents at a time to Elastic. Each document may be up to 1MB in size but are generally approx 100 - 200k. An instance of this job runs on each server.
During heavy indexing we see the CPU hit (and remain around) 500% and our throughput fall to be approx 3 documents indexed / second. I have run hotthreads but I can upload the full text as it makes the post too long and I can only upload images. Heres a snippet:
103.2% (516.2ms out of 500ms) cpu usage by thread 'elasticsearch[server3-3][management][T#5]'
3/10 snapshots sharing following 31 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
103.2% (516.2ms out of 500ms) cpu usage by thread 'elasticsearch[server3-3][management][T#4]'
3/10 snapshots sharing following 15 elements
org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:723)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:210)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:163)
103.2% (516.1ms out of 500ms) cpu usage by thread 'elasticsearch[server3-3][management][T#2]'
2/10 snapshots sharing following 31 elements
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
103.2% (516ms out of 500ms) cpu usage by thread 'elasticsearch[server3-3][management][T#3]'
4/10 snapshots sharing following 38 elements
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
101.7% (508.3ms out of 500ms) cpu usage by thread 'elasticsearch[server3-3][management][T#1]'
3/10 snapshots sharing following 16 elements
org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:539)
org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:723)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:210)
I have seen posts on here with similar hot thread traces but I didn't see an answer explaining the behvaiour. As mentioned we are only performing indexing (no searches). We are not using the completion suggestor.
While this is happening we see request timeout errors on nodes talking to each other:
[2018-06-07T13:40:42,983][WARN ][o.e.t.TransportService ] [server3-1] Received response for a request that has timed out, sent [16113ms] ago, timed out [1113ms] ago, action [cluster:monitor/nodes/stats[n]], node [{server2-1}{fEEZg21jSi63Q7Ki_4A9zA}{mjKn7-fVQgyB5MPqkunRNQ}{server2-1}{IPADDRESS:9300}], id [32784096]
We also see excessive GC times (although the memory is generally reclaimed):
[2018-06-07T14:41:51,825][WARN ][o.e.m.j.JvmGcMonitorService] [server3-1] [gc][old][496267][101] duration [14.3s], collections [1]/[15.1s], total [14.3s]/[1m], memory [22gb]->[8.9gb]/[29.8gb], all_pools {[young] [374.4mb]->[5.6mb]/[865.3mb]}{[survivor] [108.1mb]->[0b]/[108.1mb]}{[old] [21.5gb]->[8.9gb]/[28.9gb]}
[2018-06-07T14:41:51,825][WARN ][o.e.m.j.JvmGcMonitorService] [server3-1] [gc][496267] overhead, spent [14.4s] collecting in the last [15.1s]
After about 4 weeks or running we get OOM on a node (shortly followed by OOM on other nodes).
Our refresh interval is set to 60 seconds.
Using top I can see the CPU is consumed by the user column (and not sys or iowait for example).
Schema:
Schema is available on request but due to size limits I could not upload it here.
Environment details:
Elasticsearch version (bin/elasticsearch --version):
Version: 5.5.1, Build: 19c13d0/2017-07-18T20:44:24.823Z, JVM: 1.8.0_141
JVM version (java -version):
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
OS version (uname -a if on a Unix-like system):
Linux kyc-demo 2.6.32-696.10.1.el6.x86_64 #1 SMP Tue Aug 22 18:51:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
We have 9 servers each with 132G RAM. We allocate 30G to the Elastic VM. There are other processes on this box also. Each box has 16 CPUs. The CPUs are generally under utilised.
We use Spinny disks.
We have approximately 300 date based indices with a total of 1550 primary shards with 1 replica for each shard (so 3000 shards in total).
Any help will be much appreciated.