Es forcemerge question

Elastic Stack Elasticsearch
1

i use ilm to run forcemerge with max_num_segments=5 and this index have 5shard a one replication,so it have 10 shards total

      "warm": {
                "min_age": "25h",
                "actions": {
                    "forcemerge": {
                        "max_num_segments": 5
                    }
                }
            },

when i finshed it
_cat/indices see below

h     s    idx             segmentsCount p r docsCount docsDeleted storeSize pri.store.size fielddataMemory getExistsTotal memoryTotal mergesCurrent mergesTotal mergesCurrentDocs mergesCurrentSize flushTotal refreshTotal
green open xxx            97 5 1   1029288           0   798.3gb        406.4gb              0b           2906       7.8mb             0         115                 0                0b        273           20

you can see it has 97 segmentsCount more than 50
when i run /_refresh, i get below ,it change to 50 ,is it a es bug???

h     s    idx             segmentsCount p r docsCount docsDeleted storeSize pri.store.size fielddataMemory getExistsTotal memoryTotal mergesCurrent mergesTotal mergesCurrentDocs mergesCurrentSize flushTotal refreshTotal
green open xxx            50 5 1   1029288           0   783.7gb        391.8gb              0b           2906       7.6mb             0         115                 0                0b        273           30
2

Are you still writing into this index?

3

may be still writing some data，but why the segmets reduce immediately after refresh

4

I would strongly recommend against running a force merge while writing to the index, from the docs;

Force merge should only be called against an index after you have finished writing to it. Force merge can cause very large (>5GB) segments to be produced, and if you continue to write to such an index then the automatic merge policy will never consider these segments for future merges until they mostly consist of deleted documents. This can cause very large segments to remain in the index which can result in increased disk usage and worse search performance.

5

When executing forcemerge, it is best to ensure that the index is no longer written. I know this.
Our es index is one index per day . Most real-time data will not be written to yesterday’s index, but the clocks of some clients may not be synchronized, causing some data （few） to be written to yesterday’s index, and yesterday’s index has started to execute forcemerge to optimize the query performance。
i share part of our ilm settings

  "hot": {
                "min_age": "0ms",
                "actions": {
                    "set_priority": {
                        "priority": 100
                    }
                }
            },
            "warm": {
                "min_age": "25h",
                "actions": {
                    "forcemerge": {
                        "max_num_segments": 5
                    }
                }
            },
            "cold": {
                "min_age": "28h",
                "actions": {
                    "set_priority": {
                        "priority": 0
                    },
                    "allocate": {
                        "number_of_replicas": 1
                    }
                }
            },

thanks，we consider other solutions later。thanks for your reply。

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.