Merge throttling is preventing heavy bulk indexing (ES 1.7.5)

Elastic Stack Elasticsearch
1

I'm using elastic 1.7.5. My use case includes a single well provisioned server with two ES instances running on it. I am indexing events for near real-time search and aggregation. There are no updates or deletes. Events are 3K records, and I've observed up to 100Keps.

I am unable to index at that rate due to various issues with bulk indexing -- one of which is index throttling (described below). My server is limited to a RAID6 with legacy magnetic disks. The server is continuously indexing these events through a transport client using the Bulk API. Unfortunately, at this time, I am currently limited to this server, and upgrading to ES 2.0 is not an option I can currently consider. 8 shards, daily indices.

One of the issues that I am currently trying to eliminate is that elastic is itself reducing its net indexing rate by throttling. Regardless of the max_merge_count I select, I seem to encounter throttling. You'll see below that even if I bump the merge count way up to 20, I still hit this limitation almost constantly.

Disk I/O utilization reported by sar/iostat doesn't seem to increase much above 5%.

2016-03-14 11:41:18,433 [INFO ][index.engine             ] [clusterhost-1] [events-2016.03.14][0] now throttling indexing: numMergesInFlight=21, maxNumMerges=20
2016-03-14 11:41:18,583 [INFO ][index.engine             ] [clusterhost-2] [events-2016.03.14][5] now throttling indexing: numMergesInFlight=21, maxNumMerges=20
2016-03-14 11:41:18,588 [INFO ][index.engine             ] [clusterhost-1] [events-2016.03.14][0] stop throttling indexing: numMergesInFlight=19, maxNumMerges=20
2016-03-14 11:41:18,673 [INFO ][index.engine             ] [clusterhost-2] [events-2016.03.14][5] stop throttling indexing: numMergesInFlight=19, maxNumMerges=20

If I am continuously receiving data, the last thing I want is for ES to stop indexing. What's at work here? How can I support continuous indexing without hitting throttling? Are there any changes to standard merge/segment guidance when using RAID?

# magnetic disk
index.merge.scheduler.max_thread_count: 1
# increased from 8, to 16, to 20
index.merge.scheduler.max_merge_count: 20
# magnetic disk
index.store.throttle.type: none

# index settings
"index.merge.policy.max_merged_segment": "8gb",
"index.merge.policy.segments_per_tier": 20,
index.refresh_interval: 30s
index.merge.policy.max_merge_at_once: 20

Your help and expertise is appreciated.

2

Handling 100k 3kB events per seconds corresponds to around 293MB/s if I have calculated correctly. This sounds like a lot to handle for a single server with spinning disks, so I suspect you will need to scale out to reach the volumes you are targeting.

3

I agree -- 100k is a high target. Simple tests seem to suggest I get around (EDIT: 1200MB/s, was 190MB/s) under ideal conditions. From a disk perspective, I should be able to get over 300Keps. However, I'd still like to assess how much I can do with this server. It doesn't much make sense to scale out if I can't tune a single system well. I'd think I should be able to get around 60Keps without a problem -- but I'm limited between 20-30k due to this throttling issue.

References I found useful for calculating disk throughput:
https://wiki.archlinux.org/index.php/Benchmarking/Data_storage_devices#Using_dd
http://www.symantec.com/connect/articles/getting-hang-iops

4

I'm continuing to investigate this. A few things I've discovered:

  • Setting the refresh_interval to -1 to disable it, setting disable_flush to true, either independently or together does not seem to impact the frequency of merge throttling

  • Various changes to merge_floor do not seem to significantly impact the frequency of merge throttling

  • Increasing, or decreasing the number of shards may somewhat reduce or increase the frequency of merge throttling, but it seems like the duration or contracts relative to the duration of the frequency

Despite tweaking these settings, merging seems to cause throttling around every 5 minutes, almost on the dot. This gives an example of frequency of 'now throttling' over the last hour:

16:01 10
16:08 33
16:15 11
16:17 71
16:21 27
16:31 63
16:32 20
16:37 57
16:38 26
16:43 55
16:44 28
16:49 16
16:50 20

This configuration is barely touching the disk. and has a fairly low indexing rate... Why is it merging so much?

5

Another discovery: when I'm indexing at a modest rate, I don't observe the throttling issue. If I index at a very low rate or a very high rate, I get very aggressive merges. I am less surprised this happens at a high rate, but why would merge throttling happen frequently at a very low indexing rate?