Rollover Policy Not reacting to Number of Docs

Elastic Stack Elasticsearch
1

We have an index lifecycle policy which we have set to maxdocs=5 just to see what happens.

{
    "dev-policy": {
        "version": 2,
        "modified_date": "2019-12-17T10:14:34.142Z",
        "policy": {
            "phases": {
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_age": "1d",
                            "max_docs": 5
                        }
                    }
                },
                "delete": {
                    "min_age": "180d",
                    "actions": {
                        "delete": {}
                    }
                }
            }
        }
    }
}

Here is the index assigned to that policy:

{
    "access-dev00003": {
        "aliases": {
            "access-dev": {
                "is_write_index": true
            }
        },
        "mappings": {
            ...
        },
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "dev_policy",
                    "rollover_alias": "access-dev"
                },
                "number_of_shards": "1",
                "provided_name": "access-dev00003",
                "creation_date": "1576579441268",
                "number_of_replicas": "0",
                "uuid": "58hR5Nh9SqmfHEvDLkVhJg",
                "version": {
                    "created": "7030099"
                }
            }
        }
    }
}

I would expect a new index access-dev00004 to be created when the index exceeds 5 documents but this does not happen:

health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   access-dev2     _2Hp-oiMRb-bhXbsXzoftw   1   0    6611337      1423565    830.7mb        830.7mb
green  open   access-dev00003 58hR5Nh9SqmfHEvDLkVhJg   1   0         30            0    439.6kb        439.6kb
2

Also "Linked Indices" in Kibana remains stubbornly at 0.

image
image993×264 16.8 KB

3

Our template access-dev:

{
    "access-dev": {
        "order": 0,
        "index_patterns": [
            "access-dev*"
        ],
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "dev-policy",
                    "rollover_alias": "access-dev"
                },
                "number_of_shards": "1",
                "number_of_replicas": "0"
            }
        },
        "mappings": {
...
4

How long are you waiting? By default the poll interval is 10 minutes (this can be changed), so even if you index 5+ documents, it will only check every 10 minutes.

5

We've waited a few days with no rollover. We're missing the entire rollover - at least 1d should have triggered by now:

"rollover": {
    "max_age": "1d",
    "max_docs": 5
}
6

Okay let's find out why it's not rolling over, what's the output of:

GET /access-dev00003/_ilm/explain?human
7 
{
    "indices": {
        "access-dev00003": {
            "index": "access-dev00003",
            "managed": true,
            "policy": "dev_policy",
            "step_info": {
                "type": "illegal_argument_exception",
                "reason": "policy [dev_policy] does not exist"
            }
        }
    }
}

So it looks like a _ instead of - typo. To repair:

POST /access-dev00003/_settings
{
    "index": {
        "lifecycle": {
            "name": "dev-policy",
            "rollover_alias": "access-dev"
        }
    }
}

And explain?human now returns:

{
    "indices": {
        "access-dev00003": {
            "index": "access-dev00003",
            "managed": true,
            "policy": "dev-policy",
            "lifecycle_date": "2019-12-17T10:44:01.268Z",
            "lifecycle_date_millis": 1576579441268,
            "phase": "hot",
            "phase_time": "2019-12-19T15:02:27.505Z",
            "phase_time_millis": 1576767747505,
            "action": "unfollow",
            "action_time": "2019-12-19T15:02:27.505Z",
            "action_time_millis": 1576767747505,
            "step": "wait-for-follow-shard-tasks",
            "step_time": "2019-12-19T15:02:27.536Z",
            "step_time_millis": 1576767747536,
            "phase_execution": {
                "policy": "dev-policy",
                "phase_definition": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_age": "1d",
                            "max_docs": 5
                        }
                    }
                },
                "version": 2,
                "modified_date": "2019-12-17T10:14:34.142Z",
                "modified_date_in_millis": 1576577674142
            }
        }
    }
}
8

What is interesting is that I used Kibana's "Index Lifecycle Policies" UI to assign the index to the policy and it's not bringing any error but obviously not linking the index.

It's kinda confusing since it lists "indexes" attached to the policy but when you add stuff it's the "index template" and not the index you are adding...

9

I'm not as familiar with the Kibana UI side of things, but this does sound like either a bug or at least a usability issue. I would recommend opening an issue on the GitHub - elastic/kibana: Your window into the Elastic Stack repository to make this easier/better.

10

You have dev-policy in some places and dev_policy in another.

11

I've fixed that now (as I posted above) and I am still seeing no rollover.
My expectation is that a new index access-dev00004 should be created but everything is still merrily logging to access-dev00003.

Can I ask: how does ES know to create access-dev00004? Does it parse access-dev00003 and figure out the numbering system?

12

It does. However, you index name does not actually follow the pattern rollover uses. In order for rollover to work, it needs to end in -\d+ (a hyphen preceding the numbers), so you should name your index access-dev-000003 (or whatever number).

I suspect that if you check the ILM explain output it will be in an error state due to the name being invalid for rollover.

13

I think it uses index.provided_name, that is how things like ...-{now/d}-.... get saved.

I agree with the prior post, I think indices need to end in -000000 pattern.

14

OK, switched to logs-dev-00001 and, after 10min nothing and then... 14min later I see the new index being automatically created.

Thanks all!

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.