Setup one ILM merge task at a time

Elastic Stack Elasticsearch
1

Hello, I've been setting up ILM to merge and compress indices,
but when I checked on stats API

GET _stats?pretty

the result is two merge tasks run concurrently

      "merges" : {
        "current" : 2,
        "current_docs" : 155120315,
        "current_size_in_bytes" : 16481206343,
        "total" : 46901,
        "total_time_in_millis" : 19245044,
        "total_docs" : 803592602,
        "total_size_in_bytes" : 162075430014,
        "total_stopped_time_in_millis" : 0,
        "total_throttled_time_in_millis" : 6721799,
        "total_auto_throttle_in_bytes" : 86761260455
      }

Is there any way to limit the max concurrent merge task so my cluster performance won't affect by the current task, or is this expected behavior?
what if my cluster has hundred of indices running same ILM policy so those indices runs hundred merge tasks concurrently

hope I get explanation about this one, thanks in advance

2

This is the policy I applied on multiple indices, and ofc this is only a test policy, which leads to the concurrent merge tasks

PUT _ilm/policy/compression_test
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1ms",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1,
            "index_codec": "best_compression"
          },
          "set_priority": {
            "priority": 50
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      }
    }
  }
}

feel free to ask for additional information about my case, because I have no clue what's further information should I provide

3

anyone have insight about this?

4

Merges are usually triggered by indexing, but can also be a force-merge which may be triggered by ILM. However force-merges already run one-at-a-time on each node by default.

5

the stats I sent was from ILM, so does it mean that "two merge tasks" was run on two node?

6

The stats you shared were from GET _stats so they are not ILM-specific.

7

I mean, the tasks run automatically from ILM condition

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.