Index size is different from ILM index size

Elastic Stack Elasticsearch
1

Why my index size is different from ilm index size property

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   log-wlb-sysmon-2021.10.01-000014 OaDubXBDTFuwVqYjLIIcXg   1   1   87312747            0    114.4gb         57.2gb
green  open   log-wlb-sysmon-2021.09.30-000013 _ukLur8LR0aY4pUJPnGIcw   1   1   85912426            0    113.9gb         56.9gb
green  open   log-wlb-sysmon-2021.08.16-000015 EtF6odgpQGGRjQ0Nywqtxg   1   1  102154111            0    102.2gb         51.1gb
green  open   log-wlb-sysmon-2021.09.23-000019 tevWknvdRY-XwrEARoNY9g   1   1   57709948        59091     68.4gb         34.2gb
green  open   log-wlb-sysmon-2021.08.27-000017 hAGeCkqhRlCBvmwBLonjaQ   1   1   93303600            0      102gb           51gb
green  open   log-wlb-sysmon-2021.08.19-000016 mkBAnhNMReyHzw0PJwkCHg   1   1   83527835            0      102gb           51gb
green  open   log-wlb-sysmon-2021.09.07-000018 HuDxDct7RGKOcuYFXCKA8w   1   1   92614553            0    101.7gb         50.8gb

Here is my ilm policy for this indices

{
  "winlogbeat_sysmon_policy" : {
    "version" : 14,
    "modified_date" : "2021-10-06T09:06:34.667Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "rollover" : {
              "max_size" : "50gb",
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "60d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

In max size i specified size 50gb and i have one replica for each index so for one index size should be 50gb+50gb(P+R)=100gb.But is different.

2

Here is something from the docs (emphasis is mine):

ILM runs periodically, checks to see if an index meets policy criteria, and executes whatever steps are needed.

It's controlled by indices.lifecycle.poll_interval, which defaults to 10 minutes. See Index lifecycle management settings in Elasticsearch | Elasticsearch Guide [7.15] | Elastic

So when ILM ran for your index, the size of the index was above 50gb (but not equal to 50gb). That's why you have approximately a bit more than 50gb per shard.

If you want to be closer to he 50gb, you can decrease this value.

3

i already set this indices.lifecycle.poll_interval: "1m" in cluser settings.

4

So it indicates that you are getting a lot of documents every minute.

Is there anything wrong with this behavior?

5

yes,you are right
thanks @dadoonet

6

That's really not a great idea, you're going to be putting undue pressure on your cluster for very little positive outcome.

8

@warkolm

But in earlier reply the reason for decreasing this value was i was getting the index size greater than 50gb ,so i decreased this value

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.