Throttled merge

Hello,

Currently we have an index of 33TB (ES 7.8), 48 shards (16 primary, 32 replicas), two months ago we deleted a lot of documents and we noticed that one of the shards (the primary and the 2 replicas) has a very large size (5TB), looking at the detail of the index I notice that the merge has not been executed for 57 days:

  "merges": {
    "current": 0,
    "current_docs": 0,
    "current_size": "0b",
    "current_size_in_bytes": 0,
    "total": 4203366,
    "total_time": "96.1d",
    "total_time_in_millis": 8311549584,
    "total_docs": 15152680627,
    "total_size": "48.6tb",
    "total_size_in_bytes": 53517052444064,
    "total_stopped_time": "0s",
    "total_stopped_time_in_millis": 0,
    "total_throttled_time": "57.2d",
    "total_throttled_time_in_millis": 4942382248,
    "total_auto_throttle": "5mb",
    "total_auto_throttle_in_bytes": 5242880

Config:

      "index.merge.policy.reclaim_deletes_weight": "2.0",
      "index.merge.scheduler.max_merge_count": "5",
      "index.merge.scheduler.max_thread_count": "2",
      "index.merge.policy.deletes_pct_allowed": "33.0",
      "index.merge.policy.expunge_deletes_allowed": "10.0",
      "index.merge.policy.floor_segment": "2mb",
      "index.merge.policy.max_merge_at_once": "10",
      "index.merge.policy.max_merge_at_once_explicit": "30",
      "index.merge.policy.max_merged_segment": "5gb",
      "index.merge.policy.segments_per_tier": "10.0",
      "index.merge.scheduler.auto_throttle": "true",

The idea is not to force a merge, why is the merge not being executed in Elastic?

This is EOL, please upgrade as a matter of urgency as latest 7.X is 7.17 so you are a fair bit behind. I would definitely start there before digging further.

It looks like you have overridden the default settings. How did you arrive at these configs? What happens if you revert to the defaults?

Hi Christian, this is the default configuration, we do not modify it when creating the index.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.