ILM leaves empty shards of 225bytes

[Lin_Yu]

Hello,

I'm using elastcisearch v8.5 and filebeat.

My question is :

• How could i solve 0 bytes shard keep rolling over?
• How to delete 0bytes shards?
• How to set up ILM correctly?

This is the configuration of filebeat.yml

filebeat.inputs:
- type: log
  id: 10.3.250.81
  enabled: true
  paths:
    - /var/log/test/test.log
  ignore_older: 1d

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.template:
  name: test
  pattern: test-*

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0

setup.template.settings.index.lifecycle:
  rollover_alias: test

setup.ilm:
  policy_name: fourday-policy
  enabled: true
  rollover_alias: "test"

output.elasticsearch:
  hosts: ["10.3.249.33:9200"]
  index: test-%{+yyyy.MM.dd}

logging.to_files: true
logging.files:
  path: /var/log/test
  keepfiles: 1
logging.level: info

processors:
  - add_locale: ~
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I found few things:

1. A new datastream is created every day

   • test-{2023-05-12}, test-{2023-05-13} ... etc

2. If I've ILM enabled, then index name will become:

   • test-{2023-05-12-2023-05-12-00001}, test-{2023-05-12-2023-05-12-00002} ... etc

3. hot phase will re-enter:

   • if my hot-warm-delete is 1day-1day-2day, means the ageout will be 4 days, not 3 days
     hot --1day--> hot --1day--> warm --1day--> delete
     |-- new index--> hot --1day--> hot --1day--> warm --1day--> delete
     |-- new index--> hot --1day--> hot --1day--> warm --1day--> delete .....

4. 0bytes index never delete after ageout. the document have been delete, but shards still there, then the limit of 1000 shards will be full soon.

5. min_doc option not working in hot phase:

   • I've tried to set this option to stop rolling over empty index, but it's not working

6. no ways to delete the 0bytes index:

   • empty index is continuously occupied by datastream, and if delete datastrem, will also delete the index with documents.

curl -XGET "localhost:9200/_cat/indices/*test*?s=docs.count&pretty"
green open .ds-test-2023.05.10-2023.05.12-000019 vU7po9FTRVC3Q12OEWVFMw 1 0   0 0    225b    225b
green open .ds-test-2023.05.11-2023.05.12-000018 kqCKBOa_TDulM01HEDBjAw 1 0   0 0    225b    225b
green open .ds-test-2023.05.10-2023.05.12-000018 TXQhOhP_SS6F9DGbOS6_NQ 1 0   0 0    225b    225b
green open .ds-test-2023.05.11-2023.05.12-000019 DmjNhaM0SNiZ_KpIhRlEhg 1 0   0 0    225b    225b
green open .ds-test-2023.05.12-2023.05.12-000017 _6Fp-3SgTeu2tTaEunLW5A 1 0 240 0 251.3kb 251.3kb

curl -XGET localhost:9200/_data_stream/*test-2023.05.10*?pretty
{
  "data_streams" : [
    {
      "name" : "test-2023.05.10",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-test-2023.05.10-2023.05.12-000018",
          "index_uuid" : "TXQhOhP_SS6F9DGbOS6_NQ"
        },
        {
          "index_name" : ".ds-test-2023.05.10-2023.05.12-000019",
          "index_uuid" : "vU7po9FTRVC3Q12OEWVFMw"
        }
      ],
      "generation" : 20,
      "status" : "GREEN",
      "template" : "test",
      "ilm_policy" : "fourday-policy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    }
  ]
}

curl -XDELETE localhost:9200/.ds-test-2023.05.10-2023.05.12-000019?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "index [.ds-test-2023.05.10-2023.05.12-000019] is the write index for data stream [test-2023.05.10] and cannot be deleted"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "index [.ds-test-2023.05.10-2023.05.12-000019] is the write index for data stream [test-2023.05.10] and cannot be deleted"
  },
  "status" : 400
}

This is mine ILM policy

{
  "fourday-policy" : {
    "version" : 3,
    "modified_date" : "2023-05-12T00:34:05.174Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "1d",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 1
            },
            "forcemerge" : {
              "max_num_segments" : 1
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "2d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      },
      "_meta" : {
        "managed" : true,
        "description" : "test ILM policy"
      }
    },
    "in_use_by" : {
      "indices" : [
        ...
      ],
      "data_streams" : [
        ...
      ],
      "composable_templates" : [
        "test"
      ]
    }
  }
}

[BenB196]

As of Elasticsearch 8.5, ILM will no longer rollover empty indices Prevent ILM from spuriously rolling over (many) empty indices · Issue #86203 · elastic/elasticsearch · GitHub & ILM don't rollover empty indices by joegallo · Pull Request #89557 · elastic/elasticsearch · GitHub.

If you're using ILM, I think you should set your output index to the alias and not date math:

Currently you have:

setup.ilm:
  policy_name: fourday-policy
  enabled: true
  rollover_alias: "test"

output.elasticsearch:
  hosts: ["10.3.249.33:9200"]
  index: test-%{+yyyy.MM.dd}

I think you can do:

setup.ilm:
  policy_name: fourday-policy
  enabled: true
  rollover_alias: "test"

output.elasticsearch:
  hosts: ["10.3.249.33:9200"]
  index: test

Note though that ILM will always rollover an index that meets its other conditions (age, size, doc count) if it has at least 1 document in it (This will always leave, 1, index with 0 docs if you are no longer writing to that index). If you don't want ILM to rollover on small indices, take a look the min_* settings for ILM options.