ILM keep rolling over empty indexes

Adam_Lin · May 4, 2023, 1:20am

Hi

according to the following release note, since 8.5.3, ILM won't rollover the empty index by default.

github.com/elastic/elasticsearch

ILM don't rollover empty indices

elastic:main ← joegallo:ilm-dont-rollover-empty-indices

opened 04:56PM - 23 Aug 22 UTC

joegallo

+278 -123

Closes #86203, related to #89283 ILM will no longer roll over empty indices b…y default. This is accomplished by injecting a `min_docs: 1` condition into the rollover dry run request in `WaitForRolloverReadyStep`. Such a condition will only be injected if there's not already a `min_docs` (or `min_primary_shard_docs`) condition associated with the rollover, so a policy can opt out of the new behavior by adding an explicit `min_docs: 0` condition to the rollover action. This can also be disabled on a cluster-wide basis by setting `indices.lifecycle.rollover.only_if_has_documents` to `false`.

I'm using the elk server v8.6.0, and with the ILM policy

{
  "testpolicy" : {
    "version" : 2,
    "modified_date" : "2023-05-02T06:46:02.069Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "1h",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 1
            },
            "forcemerge" : {
              "max_num_segments" : 1
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1h",
              "min_docs" : 1
            }
          }
        },
        "delete" : {
          "min_age" : "3h",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      },
      "_meta" : {
        "managed" : true,
        "description" : "Test ILM policy"
      }
    },
    "in_use_by" : {
      "indices" : [
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000022",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000024"
      ],
      "data_streams" : [
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03",
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02",
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01"
      ],
      "composable_templates" : [
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======"
      ]
    }
  }
}

after few days, the empty indexes still stay there.

green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002 wJEpsDXEQqaJ-dRPbu4hwg 1 0   0 0    225b    225b
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000022 GWNx3vOhSj-G74SxG7OCcQ 1 0 561 0 446.3kb 446.3kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028 Zxc6LYWhRGanBl5F3lSvNw 1 0 399 0 416.1kb 416.1kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040 mALAWq6wTMuqC8C769MOsg 1 0   0 0    225b    225b
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026 K8QRwp39SuuJFK8SJzksrg 1 0 561 0 534.4kb 534.4kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000024 gg_9GVgWS1WXckdKVQUW8w 1 0 564 0 449.1kb 449.1kb

I wish that the age-out will be going to the warm phase after an hour and deleted after 3 hours.
There is no problem with ILM within one day, and it scrolls normally, but on the next day, a new index is created, and the old index stays in the HOT stage

for example, the index ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002" is already age-out, but when I check the stat this index, it still stops in the hot phase.

{
  "indices" : {
    ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002" : {
      "index" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
      "managed" : true,
      "policy" : "testpolicy",
      "index_creation_date_millis" : 1683013920597,
      "time_since_index_creation" : "1.72d",
      "lifecycle_date_millis" : 1683013920597,
      "age" : "1.72d",
      "phase" : "hot",
      "phase_time_millis" : 1683013922027,
      "action" : "rollover",
      "action_time_millis" : 1683013922443,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1683013922443,
      "phase_execution" : {
        "policy" : "testpolicy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1h",
              "min_docs" : 1
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1683009962069
      }
    }
  }
}

this is my indexes and datastreams:

{
  "index_templates" : [
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "index_template" : {
        "index_patterns" : [
          "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "testpolicy",
                "rollover_alias" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======"
              },
              "mapping" : {
                "total_fields" : {
                  "limit" : "10000"
                }
              },
              "refresh_interval" : "5s",
              "number_of_shards" : "1",
              "max_docvalue_fields_search" : "200",
              "query" : {
                "default_field" : [
				   ...
                ]
              },
              "number_of_replicas" : "0"
            }
          },
          "mappings" : {
		   ...
          }
        },
        "composed_of" : [ ],
        "priority" : 150,
        "data_stream" : {
          "hidden" : false,
          "allow_custom_routing" : false
        }
      }
    }
  ]
}

{
  "data_streams" : [
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
          "index_uuid" : "wJEpsDXEQqaJ-dRPbu4hwg"
        }
      ],
      "generation" : 3,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    },
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040",
          "index_uuid" : "mALAWq6wTMuqC8C769MOsg"
        }
      ],
      "generation" : 43,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    },
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026",
          "index_uuid" : "K8QRwp39SuuJFK8SJzksrg"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028",
          "index_uuid" : "Zxc6LYWhRGanBl5F3lSvNw"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000030",
          "index_uuid" : "fAxgWI5nRkKXJnCqMHhY3g"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000032",
          "index_uuid" : "8V2WJXDeT4CU7OzZZ8gNYw"
        }
      ],
      "generation" : 32,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    }
  ]
}

Could someone help to correct me if there have configuration errors or any stuff I misunderstood?

Thanks.

Adam_Lin · May 5, 2023, 1:16am

Hello

anyone could help?

Adam_Lin · May 8, 2023, 7:28am

Could anyone help for this?

Tyler_Perkins · May 26, 2023, 1:52pm

Hi Adam,
Sorry you're having trouble with ILM. With that feature in 8.5 we are trying to prevent empty indices which have overhead in the cluster and no value.

If I understand correctly, you want to use this feature and avoid empty indices, but you still want to rollover on your expected schedule. If so, I would suggest to try removing the "min_docs" : 1 from the Hot phase in your ILM policy and see if you get the expected behavior. I think the distinction here is between 'rollover' and 'index creation or not'. I think your usage of min_docs is preventing ILM to move to the next phase if there isn't at least one doc, whereas without that parameter it can move on even if there were no docs (but there will not be an empty index).

If the documentation for this is unclear please let us know.

system · June 23, 2023, 1:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.