Help on understanding indexing rate plots for different configurations (logstash batch size, nb of shards, nb of disks)

[anon24174512]

Hello,

I am trying to understand my plots of the impact of some configuration parameters on the indexing rate of my testing ELK cluster. I plotted in the linked pictures the average (over 5 iterations) of the number of indexed documents per second for each combination of parameters and the average indexing times in the second plot.

The cluster setup is four machines with 32Gb, Intel Core i7 3.60GHz, 2 HDD.
Zookeeper : node 1, node 2
Kafka : node 1, node 2
Elasticsearch: node 3, node 4
Logstash : node 3, node 4
Kibana: node 1

Parameters range:
Logstash batch size values: 2500 5000 10000 15000 20000 40000
ES number of shards: 1 2 3 4 5 6 8 16 32
ES number of replicas: 1
Disks: one disk or two disks stripped in ES config

Three methods are used in the plots to measure the indexing rate:

1. The average measured indexed documents/s is done by sampling each 5s with _cat/count/${index}* and computing the average once the number of documents to index is reached.
2. The average number of indexed documents/s from primaries is computed with ${index}*/_stats/indexing by dividing the primaries count by the primaries indexing time.
3. Same as 2 but for all.

There are 2877306 documents to index. The average size of a document is ~440 bytes. It contains around 20 fields which are parsed by a template.

Questions when looking at the average indexing time:

1. How the measured times using _cat/count/${index}* (method 1) can be lower than those from ${index}*/_stats/indexing on primaries (method 2)? Are there some specificities when using count or stat on primaries on how the time spent for indexing is counted ?
2. Why when using two disks with ES stripping there is no improvement (even worst indexing time)?

Questions where the answers probably depend on question 1:

1. Why the measured times become lower than those from primaries (which stay stable) as the logstash batch size is increased?
2. Indexing time on primaries improved a little or stabilize as the number of shards increase with large logstash batch size. On measured indexing time, the trends is the same but it stays higher than with one shards. Any clue why ?