About segment merge

Elastic Stack Elasticsearch
1

Hello, I have a heavily indexed elasticsearch cluster, about 20K lines per second, and one index per day.
Each index has about 300 segments. For example, segment info of some index (2017-08-19) is partially list below:

Blockquote
index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound
qn_2017-08-19 0 r 10.2.93.185 _2vsr 134523 35865749 0 5gb 9752042 true true 6.5.1 false
qn_2017-08-19 0 r 10.2.93.185 _4n3y 216574 34562168 0 5gb 9048328 true true 6.5.1 false
qn_2017-08-19 0 r 10.2.93.185 _637c 284088 34114706 0 5gb 9499344 true true 6.5.1 false
qn_2017-08-19 0 r 10.2.93.185 _661v 287779 2481838 0 380.7mb 791589 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _68fs 290872 1336373 0 198.7mb 419710 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6af6 293442 828156 0 121mb 279891 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6c1p 295549 1731130 0 254.9mb 515564 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6eaf 298455 942425 0 137.2mb 285555 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6gp9 301581 2502974 0 373.1mb 748680 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6j0o 304584 385143 0 56.9mb 146525 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6jqq 305522 927247 0 134.6mb 268981 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6ly0 308376 1776647 0 259.9mb 493364 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6md9 308925 76160 0 11.4mb 42011 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6mgj 309043 294887 0 43.3mb 108513 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6mwz 309635 243215 0 34.6mb 102228 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6nd4 310216 118901 0 17.7mb 54620 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6nkl 310485 248982 0 36.1mb 110568 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6nvy 310894 126586 0 18.4mb 58437 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6o3s 311176 135339 0 19.8mb 62553 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6ocz 311507 273520 0 39.9mb 103114 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6onk 311888 72061 0 10.7mb 38558 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oqn 311999 53366 0 8mb 34997 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oto 312108 100517 0 14.9mb 45692 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oy4 312268 44420 0 6.7mb 30418 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oz4 312304 34652 0 5.2mb 27141 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oz5 312305 3 0 5.6kb 2466 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oz6 312306 10 0 8.8kb 4248 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oz7 312307 1 0 5.6kb 2706 true true 6.5.1 true
qn_2017-08-19 0 r 10.2.93.185 _6oz8 312308 1 0 5.5kb 2825 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _2vsr 134523 35865749 0 5gb 9752042 true true 6.5.1 false
qn_2017-08-19 0 p 10.2.93.205 _4n3y 216574 34562168 0 5gb 9048328 true true 6.5.1 false
qn_2017-08-19 0 p 10.2.93.205 _637c 284088 34114706 0 5gb 9499344 true true 6.5.1 false
qn_2017-08-19 0 p 10.2.93.205 _661v 287779 2481838 0 380.7mb 791589 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _68fs 290872 1336373 0 198.7mb 419710 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6af6 293442 828156 0 121mb 279891 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6c1p 295549 1731130 0 254.9mb 515564 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6eaf 298455 942425 0 137.2mb 285555 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6gp9 301581 2502974 0 373.1mb 748680 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6j0o 304584 385143 0 56.9mb 146525 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6jqq 305522 927247 0 134.6mb 268981 true true 6.5.1 true
qn_2017-08-19 0 p 10.2.93.205 _6ly0 308376 1776647 0 259.9mb 493364 true true 6.5.1 true

As we can see, there are some very small segments. However, after I trigger a force merge, no merge is done.
My question is why force merge is not done?
Thanks a lot :smiley:

2

How did you run the force merge? Did you set max_num_segments to 1?

3

yes, I tried different max_num_segments, for example 1, 10, 100.
Doesn't work.

4

Which version of Elasticsearch are you on? Is there anything in the logs?

5

My elasticsearch is 5.4.3, nothing special in my log.
When trigger a force merge, curl return immediately.

Blockquote
curl -XPOST http://ip:port/qn_2017-08-19/_forcemerge
{"_shards":{"total":10,"successful":10,"failed":0}}

I tried to do force merge other indices which are very small only containing thousands of docs, the force merge works. And curl waits until merge is done.

6

This issue confuses me for a long time.
I tried to do force merge to improve search performance and save some file handles.
I google misc docs, such as index throttle, merge scheduler. Looks like they are not related to my issue.

7

In your example you are not specifying max_num_segments. If you do this on the index you showed before, how long does it take for it to return?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.