Index is not rolling

Elastic Stack Elasticsearch
1

Hi,

I have the filebeat policy

image
image808×93 5.29 KB

The hot phase is configured

image

and the delete phase in the pilicy is set to

image

But the index size is growing daily.

image
image900×35 4.56 KB

And the status tell me

image

The policy is assigned to the index template too

image

Why is this so?

2

Hi @oicfar,

based on your post, it looks like it does at least rollover every day or every second day. Can you clarify that? If you have very rapid growth, the index might grow above the size setting, but should then rollover shortly after (by default 10 minutes).

I would advice to try the ILM explain API on the index to see if it contains useful information.

And double check that the setting indices.lifecycle.poll_interval is still set to its default of 10 minutes.

3

Correct, the index is not rolling every day. And I found, that the new policy is not set for the index. Cause:

curl -X GET "localhost:9200/filebeat-7.8.1-2020.07.27-000001/_ilm/explain?pretty"
{
  "indices" : {
    "filebeat-7.8.1-2020.07.27-000001" : {
      "index" : "filebeat-7.8.1-2020.07.27-000001",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1595861869294,
      "age" : "7.92d",
      "phase" : "hot",
      "phase_time_millis" : 1595861869637,
      "action" : "rollover",
      "action_time_millis" : 1595862436410,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1595862436410,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1594660050819
      }
    }
  }
}

In the response you se the initial values. I changed them in the policy.

But when I try assign the filebeat policy again I get this.

image
image724×438 27.2 KB

Should be "Alias for rollover index" set? If yes, which name should be set?

4

Hi @HenningAndersen,

today I deleted the index manually. My hope was, that the new index will get the new policy settings.

And ...

$ curl -X GET "localhost:9200/filebeat-7.8.1/_ilm/explain?pretty"
{
  "indices" : {
    "filebeat-7.8.1" : {
      "index" : "filebeat-7.8.1",
      "managed" : false
    }
  }
}

Why is this so? The policy is assigned to the index template

image

Regards,
Rafal

5

Hi @oicfar,

the new index filebeat-7.8.1 does not match the pattern (missing the -* part). But really to use ILM, you need filebeat-7.8.1 to be an alias pointing to a write index. There is more information on this here:

The bootstrap must be performed correctly for ILM to work. Work has been done to improve this in the form of a future concept called data streams.

Concerning

Should be "Alias for rollover index" set? If yes, which name should be set?

Yes, this needs to be set, but the screenshot does not state that it is not set. You have to go to the index template to check that it has something like below in its settings:

{
  "index": {
    "lifecycle": {
      "name": "filebeat",
      "rollover_alias": "filebeat-7.8.1"
    },
7

Hi @HenningAndersen,

I deleted the Index and recreated it. Just to see what happens.

Current status:

  "settings": {
    "index": {
      "lifecycle": {
        "name": "filebeat",
        "rollover_alias": "filebeat"
      },

image

And this looks better

image

image

Now I have wait 2 days for rolling and 4 days for delete.

8

Today it looks better when I check the ILM parameters.

$ curl -X GET "localhost:9200/filebeat-7.8.1-2020.08.05-000001/_ilm/explain?pretty"
{
  "indices" : {
    "filebeat-7.8.1-2020.08.05-000001" : {
      "index" : "filebeat-7.8.1-2020.08.05-000001",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1596642651766,
      "age" : "19.34h",
      "phase" : "hot",
      "phase_time_millis" : 1596646785724,
      "action" : "rollover",
      "action_time_millis" : 1596643185906,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1596646785724,
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 3,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "4gb",
              "max_age" : "2d"
            },
            "set_priority" : {
              "priority" : null
            }
          }
        },
        "version" : 12,
        "modified_date_in_millis" : 1596625189765
      }
    }
  }
}

I hope, that the rollover will work tomorrow.

9

After upgrade to 7.9.0 I get some problems. Than I deleted all configs for 7.8.0 and 7.8.1 and now it's looks good.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.