Slow down/throttle indexing

Elastic Stack Elasticsearch
1

I have a 3 node (all data/master) cluster running version 2.3.1. Each day I build a new 70GB (includes 1 replica) index, switch an alias at the end and drop the old index. The issue I'm having is the build process is hurting search performance. It gets worse at the end when I re-enable refresh (set to -1 for build phase).

I'd like to be able to throttle the index so it won't impact searching so much. so far I've tried:

Setting the refresh to 30s so it's not one big hit at the end (helped a little)
Playing with indices.store.throttle.max_bytes_per_sec (and type=merge), didn't seem to make any difference)

Any ideas?

Many thanks

2

Those settings won't do anything in 2.X as throttling is automatic - https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

How do you know it's throttling?

3

Those settings won't do anything in 2.X as throttling is automatic

OK, that makes sense - not sure how the link you provided relates though (I don't use logstash)?

Changing the refresh rate to 30s throughout the build did seem to help, in that there was not a big refresh right at the end of the build where I was setting it back from -1 to 1s.

4

Oh, wrong link :frowning:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/breaking_20_setting_changes.html#_merge_and_merge_throttling_settings

5

:slight_smile: thank you, that makes more sense. It looks like I'm not able to throttle this from the ES end. I guess I'll have to look at slowing it down from the source.