ILM keeping rolling over the same old indexes daily

Elastic Stack Elasticsearch
1

Hi,

I have fluentd pushing logs into elasticsearch with index names based on the date, e.g. logs.kubelet.YYYY.MM.DD and using index lifecycle management (ILM). I have set the ILM policy to roll over after 50GB or daily, whichever comes first. What is strange is that it seems to continue to roll over old indexes as the days go by even though no new documents are added to them.

For example, an index from 7 days ago has been rolled over from suffix -000001 all the way up to -0000007. The indexes with suffixes -000002 and up are all empty.

Does anyone have any idea where I might have messed things up?

Here's the ILM policy doc:

{
  "fluentd-logs" : {
    "version" : 413,
    "modified_date" : "2020-10-07T22:48:10.328Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "14d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  }
}
2

When you are using rollover you need to index into the write alias and not date based indices like you do now. You should update your fluentd index to write to the write alias, which will allow rollover to work properly. Now it probably rolls over based on time as no data is written to it.

1 Like
3

I think fluentd creates a new write alias every day. So each day there's a new index, rollover indexes, write alias, index template. At least, it is supposed to do that.

I guess what you are saying here is that the ILM policy continues to apply even though no new documents are being indexed.

I suppose if I disabled the max_age then it would stop rolling over to a new index, but then it would never delete the "last" old index because the delete action only applies after the index rolls over.

I guess I have to sacrifice having the date in the index names if I want to use ILM, because the ILM policy will never delete the "current" write alias.

4

No, that is not how rollover works. I do not believe fluentd has any support for rollover so it likely just creates one index per day alongside the rollover index you have configured.

I suspect the ILM policy may not apply at all to the indices created by fluentd.

Does not matter if no data is indexed into it as per my previous point.

You do not need this anyway as ILM base phase logic on index metadata.

5

I wonder if it would make sense to have a feature request / bug report not to rollover empty indexes ? It seems like a waste to do so.

6

The fluentd elasticsearch plugin has added some ILM support in recent months, so it does actually create a new index template, rollover index, and so on for each day if we configure it that way.

However, it seems if we configure it this way, we shouldn't use age based rollover in the ILM policy. It should only be used to set a maximum size for the indexes and we would have to use curator to clean up indexes after a period of time, which seems to limit the usefuless of ILM.

I'll migrate over to not having the date in the index name, it seems like the right thing to do now that I've realized how the rollover logic works.

7

I was not aware of that. It does however sound like a very odd way of using rollover as that is not how it was designed to work as far as I know.

8

I wouldn't blame the plugin author, it's just possible using some combination of options that perhaps I should not have used together.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.