I've got 2 similar deployments of our application:
5.6.1 elasticsearch, soon to be upgraded
VM virtualisation
8 CPU (Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz)
12 Gigs of RAM assigned
No SSD
i've got very different indexing rate on both machines, with the same data to index.
the fast machine needs about 0.2 ms to index one document, the slow one 0.9ms, and has usually a very high load. i think it's an issue of the Virtualisation but i just want to rule out, that it's the hard disk. So on both machines, top shows me wa(it) usually 0, sometimes up to 2. I think it's not a harddrive issue. if it would be i'd expect much higher waits. Or am I wrong?
i calculate the time needed to index 1 doc by stats:
node.indices.indexing.index_time_in_millis / node.indices.indexing.index_total
iostat sample
iostat 5
Linux 3.10.0-693.2.2.el7.centos.plus.x86_64 (tracer2) 2018-10-02 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
35,51 22,29 2,01 0,22 0,00 39,96
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 181,99 125,30 11490,17 129009633 11830256952
avg-cpu: %user %nice %system %iowait %steal %idle
43,15 50,56 2,83 0,30 0,00 3,16
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 322,60 0,00 14434,40 0 72172
avg-cpu: %user %nice %system %iowait %steal %idle
52,43 41,70 2,73 0,03 0,00 3,11
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 283,40 0,00 5685,60 0 28428
avg-cpu: %user %nice %system %iowait %steal %idle
52,77 43,05 2,36 0,03 0,00 1,80
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 320,40 0,00 31557,60 0 157788
avg-cpu: %user %nice %system %iowait %steal %idle
49,03 44,37 2,41 0,40 0,00 3,79
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 298,00 0,00 7840,80 0 39204
avg-cpu: %user %nice %system %iowait %steal %idle
50,03 45,94 2,20 0,03 0,00 1,80
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 316,80 0,00 15043,20 0 75216
avg-cpu: %user %nice %system %iowait %steal %idle
50,66 44,10 2,53 0,03 0,00 2,68
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 275,60 0,00 8480,80 0 42404
can i rule out the Harddisk as reason for this slow index speed?
thanks!