Index rollover due to policy does not copy mapping

Elastic Stack Elasticsearch
,
1

Hello,

I have setup an index with a 'date' field in milliseconds (epoch_millis), I did set this explicitly while creating the index. Then I attached a policy to rollover after x days.

After x days, a new index is created but Kibana is not showing it due to wrong mapping or missing 'date' field. When I checked the new auto created index, it does not have 'date' field mapped as Date.

How can I ensure that the new auto generated indices by the lifecycle policy also copies the same field mappings or atleast the 'date' field from previous or parent index?

Thanks

2

You need to specify the mapping through an index template.

3

Thanks for the reply.

Index template for the main/primary index or for the new ones created by the policy? Can you elaborate?
I have setup an index template but it only allows to set field as Date, how do I specify format which is epoch_millis in my case?

4

I think you can create a template like the following:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "your_index_ilm_policy",
          "rollover_alias": "your_index"
        }
      }
    },
    "mappings": {
      "properties": {
        "someDateField": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        }
        "anotherField": {
          "type": "keyword"
        }
      }
    },
    "aliases": {}
  }
}

And then for every your_index-00000* index created by the ilm policy the mapping defined in the template will be applied to the new index

1 Like
5

Thanks, I will try that.
My goal is to clear old data after X days, can I achieve this within same index without going for rollover or creating new indices?
I tested the policy without rollover and only setting up delete phase, but it deleted the index itself after the specified time.

6

Not with ILM, ILM works on entire indices.

To delete data on an index after some specific time you would need to manually execute a delete_by_query request.

The best approach is to use ILM with rollover or if you do not want to use the rollover on ILM you would need to create new indices yourself, like daily or monthly indices of example.

2 Likes
7

Thanks for the reply.
If I have an index named index-0001 setup for a rollover after 30 days and that would create index-0002, I have also setup Delete at 7 days.
When will delete occur and on which index?

If I don't setup Delete and only use Rollover, can I manually delete index-0001 once index-0002 is created and being used for writing?

Thanks

8

If you are using rollover with ILM the next pahse is based on the rollover date. If your index is set with specific size and max age of 30 days it may rollover after 30 days. If your delete phase is set to 7 days, the index no longer written to will be deleted after an additional 7 days. You will therefore hold up to 37 days worth of data in these indices. Note that if rollover happened earlier due to size the period would be less.

1 Like
9

I defined a template:

{
  "template": { 
    "settings": {
      "index": {
        "lifecycle": {
          "name": "index-rollover-policy",
          "rollover_alias": "my-index"
        },
        "number_of_replicas":0
      }
    },
    "mappings": {
      "properties": {
        "date": {
          "type": "date",
          "format": "epoch_millis"
        }
      }
    },
    "aliases": {}
  },
  "index_patterns": [
    "my-index"
  ]
}

but the index-000001 created by lifecycle policy is still showing date field type as 'text' in the mapping.
I can push new data to it, it shows doc counts increasing but I can't render the data in Kibana.
Secondly, as you see I set replica:0 in the template but the new index shows 1 replica while no policy attached to itself.

10

An index template is only applied when the index is first created, so will not apply to already existing indices. Force a rollover and verify that the index template applies to the new index that was created.

11

I understand that, the new index-000001 was created after defining the index-template.
That is why I am wondering why it didn't get the specified mapping and no policy attached to it?

12

Does the index pattern specified in the template match the name of the created index?

1 Like
13

No, the index pattern I thought is the alias which I have for the primary index. IT is without an asterisk *. I kept it same as rollover-alias.
Does it need to have a * in the index-template script so that it applies to every subsequent indices created by the policy?

14

If your indices are named e.g. myindex-000001 and so on, the index pattern must be myindex-* so it matches all created indices. It does not matter what the rollover write alias name is.

1 Like
15

Thanks for reply.
An observation:
I created a template with index pattern index-* and replicas : 0 and mapping for date field. Then I created an index manually with name index-0, it got the mapping and replicas set as 0 from the template, but... Kibana didn't render the date field as regular date time, instead showed a long numeric value (milliseconds) in date column.

16

Setting up an index template with proper mapping, alias and index pattern solved the rollover index 'date' mapping issue. Now I am waiting for auto delete of old index.
Thank you all!

1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.