Monitoring whether the index is merging at a moment

Elastic Stack Elasticsearch
1

Is the changes in number of segments per shard a good indication that merging happened (is happening)?
Is there a way to monitor when merging happens. I am running some online indexing, and it looks like merging happens quite often.

I did curl -XGET 'http://localhost:9200/files/_segments?pretty' | grep num_search_s
several time back-to-back and got different results each time?
https://gist.github.com/ja-ilija/44c6a3093db39b9f9a3d

Thanks,
Ilija

2

That is a decent indicator, you could also try turning up logging for merges (I think you can do this).

Is this a problem or are you just interested?

3

Hi,
Thanks for the answer. It is a problem that we are facing. We are indexing a rather large number of files (few hundreds a second) and our queries become somewhat slow as many threads a re created and pooled (profiling showed 40% of time is spent waiting for the available thread). Since we put the refresh rate to 1s we thought that indexing is taking a bunch of resources (and considers a bulk indexing approach). However, some of our tests showed our queries were not speeding up when indexing was truned off, so one of the suspects was that merging may be going on and that this is taking resources when indexing is off.

Will probably try to play around with trottle size to see what happens.

4

Merging is triggered by indexing documents (unless explicitly called through the API). If you disabled your indexing then ES will not be performing merges either.

It might be worth looking at the hot_threads output when you experience these problems to help determine what's causing the slow queries.

HTH

5

Thanks. What I think is happening is that the merging starts and is still
going after the indexing is suspended. I'll check the hot_threads to see
what is happening.

2015-07-10 15:16 GMT+07:00 Colin Goodheart-Smithe <
noreply@discuss.elastic.co>: