Elasticsearch indexing performance io wait

Elastic Stack Elasticsearch
1

I've got 2 similar deployments of our application:

5.6.1 elasticsearch, soon to be upgraded
VM virtualisation
8 CPU (Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz)
12 Gigs of RAM assigned
No SSD

i've got very different indexing rate on both machines, with the same data to index.

the fast machine needs about 0.2 ms to index one document, the slow one 0.9ms, and has usually a very high load. i think it's an issue of the Virtualisation but i just want to rule out, that it's the hard disk. So on both machines, top shows me wa(it) usually 0, sometimes up to 2. I think it's not a harddrive issue. if it would be i'd expect much higher waits. Or am I wrong?

i calculate the time needed to index 1 doc by stats:

node.indices.indexing.index_time_in_millis / node.indices.indexing.index_total

iostat sample

iostat  5
Linux 3.10.0-693.2.2.el7.centos.plus.x86_64 (tracer2) 	2018-10-02 	_x86_64_	(8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35,51   22,29    2,01    0,22    0,00   39,96

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             181,99       125,30     11490,17  129009633 11830256952

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          43,15   50,56    2,83    0,30    0,00    3,16

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             322,60         0,00     14434,40          0      72172

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          52,43   41,70    2,73    0,03    0,00    3,11

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             283,40         0,00      5685,60          0      28428

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          52,77   43,05    2,36    0,03    0,00    1,80

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             320,40         0,00     31557,60          0     157788

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          49,03   44,37    2,41    0,40    0,00    3,79

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             298,00         0,00      7840,80          0      39204

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          50,03   45,94    2,20    0,03    0,00    1,80

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             316,80         0,00     15043,20          0      75216

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          50,66   44,10    2,53    0,03    0,00    2,68

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             275,60         0,00      8480,80          0      42404

can i rule out the Harddisk as reason for this slow index speed?

thanks!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.