Elastic Cloud Kibana Index Lifecycle Policies is not working

Robin_Guo · April 24, 2019, 11:18am

Dear Elastic team members，

Today I did an Index Lifecycle policies on index metricbeat-2019 on Kibana. But it's not working.
Before I posted the issue here, I want to talk about a little bit of backgrounds of use case I used.

The Elasticsearch cluster I used is hosted on the elstic cloud.

Here is the license I have
(redacted)

The Cluster Deployments I used is I/O Optimized not Hot-Warm Architecture because of the trial version. Here is a cluster deployment with the details as below:

Configuration of Index Lifecycle Policies I made here:

Index metricbeat template as here:

{
  "metricbeat" : {
    "order" : 1,
    "index_patterns" : [
      "metricbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "metricbeat_ILP",
          "rollover_alias" : "metricbeat-alias"
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "50000"
          }
        },
        "refresh_interval" : "60s",
        "number_of_shards" : "5",
        "translog" : {
          "flush_threshold_size" : "4G",
          "sync_interval" : "60s",
          "durability" : "async"
        },
        "merge" : {
          "scheduler" : {
            "max_thread_count" : "1"
          },
          "policy" : {
            "max_merge_at_once_explicit" : "30",
            "max_merge_at_once" : "10",
            "max_merged_segment" : "5gb",
            "floor_segment" : "2mb"
          }
        },
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : {
      "metricbeat-alias" : { }
    }
  }
}

The problem I had here:

bhavyarm · April 24, 2019, 11:54am

Hello Robin,

Can you please let us know what exactly is not working?

Thanks,
Bhavya

Robin_Guo · April 24, 2019, 1:49pm

Hi @bhavyarm,

Could you Please advise?
It's better if we can talk over zoom or Skype.

Thanks in advance

bhavyarm · April 24, 2019, 1:58pm

Hello Robin,

Can you please raise a support ticket if you are our customer?

Thanks,
Bhavya

Bill_McConaghy · April 24, 2019, 2:07pm

Robin:

Looks like you need to set one index as the write index for that alias. Documentation for that operation is here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html#aliases-write-index

Robin_Guo · April 24, 2019, 2:32pm

Hi @bhavyarm, @Bill_McConaghy

Index lifecycle error

illegal_argument_exception: index.lifecycle.rollover_alias [metricbeat-alias] does not point to index [metricbeat-2019]

the config i made as below:

GET /_alias/metricbeat

{
  "metricbeat-2019" : {
    "aliases" : {
      "metricbeat" : {
        "is_write_index" : true
      }
    }
  }
}

Bill_McConaghy · April 24, 2019, 2:45pm

Did you retry the policy? It will not retry on its own.

Robin_Guo · April 24, 2019, 3:38pm

Hi @Bill_McConaghy,

At logstash side, I set ouput index to elasticsearch pattern like this:

metricbeat-%{YYYY}

Which means that logstash will have been sending logs to index metricbeat-2019 on elasticsearch.
even index metricbeat-2019 is putting on warm, but this index is still opening and receiving data. and a new generate index metricbeat-002020 with 0 docs. How come?

The index lifecycle policies understand should be as follows:

An index with ILP(Index lifecycle Policies) should move from hot -> warm according to index size or days we set. for the index that moves to warm should be read-only. and a new index will be generated on hot nodes.
on my case, seems the new index metricbeat-002020 is made, but there is no data is written to it.
and the data is still sent to metricbeat-2019.
from Index lifecycle management I can see Action status

By the way, Is there a best practices or good documentation I can follow step by step?

Thanks in advance

Bill_McConaghy · April 24, 2019, 3:46pm

I'm moving this to the Elasticsearch topic, as it really is more a question about how the ES ILM feature works.

gbrown · April 24, 2019, 4:29pm

Hi Robin, I'll do my best to help clear things up.

At logstash side, I set ouput index to elasticsearch pattern like this:
metricbeat-%{YYYY}
Which means that logstash will have been sending logs to index metricbeat-2019 on elasticsearch.
even index metricbeat-2019 is putting on warm , but this index is still opening and receiving data. and a new generate index metricbeat-002020 with 0 docs. How come?

When using ILM policies with Rollover (which is the action that generates the new index), your data sources should be configured to write to the alias that you have defined - in this case, that would be metricbeat from looking at your post above with the output of GET /_alias/metricbeat. When ILM creates the new index, it changes the alias so that writes to the alias are directed to the new index.

Does that make sense?

Also, I see this statement:

for the index that moves to warm should be read-only.

But that won't happen with the policy as you have it defined. You'll need to add a Read-Only action to your policy if you want indices in the Warm phase to be set as read-only. (though note indices are also set as read-only as part of the Force Merge or Shrink actions as well)

Finally, as for:

By the way, Is there a best practices or good documentation I can follow step by step?

You've probably already seen the Getting Started guide in the Elasticsearch docs, but Metricbeat (and other Beats) 6.7 and higher have built-in support for ILM, and so does Logstash when they're writing to a cluster that supports it.

If you're looking for a step-by-step guide, you may find this blog post on implementing a hot-warm architecture with ILM useful - even if you're not looking at hot-warm specifically, it should be pretty easy to adapt to whatever policy you want to use.

Does that help?

Robin_Guo · April 24, 2019, 5:26pm

Hi @gbrown,
Thanks for your detailed explanation. that helps me a lot.

I Have another 2 questions related to Index Lifecycle Policies

1. Later if we'll deploy an elasticserch cluster with hot-warm architecture on the elastic cloud. Are those 2 options enabled automatically as below?

bin/elasticsearch -Enode.attr.data=hot
bin/elasticsearch -Enode.attr.data=warm

2. I want to associate the index template with the index

Make index template

put _template/metricbeat
{
    "order" : 2,
    "index_patterns" : [
      "metricbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "metricbeat_test1",
          "rollover_alias" : "metricbeat"
        },
        "refresh_interval" : "60s",
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : {
      "metricbeat" : { 
	  "is_write_index" : true
	  }
    }
}

Check index template

GET _template/metricbeat

{
  "metricbeat" : {
    "order" : 2,
    "index_patterns" : [
      "metricbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "metricbeat_test1",
          "rollover_alias" : "metricbeat"
        },
        "refresh_interval" : "60s",
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : {
      "metricbeat" : { }
    }
  }
}

I didn't find "is_write_index": true is added to aliases block of the index template.

Do you know why?

We'll embrace with Elastic cloud on prod soon with Platinum support.

all in all, Our final aim make ILP automated without manual intervention after we defined the good policies.

Here is early morning in china, I will test IDP by follow your instruction tomorrow afternoon.

Thanks in advance

gbrown · April 24, 2019, 5:51pm

Later if we'll deploy an elasticserch cluster with hot-warm architecture on the elastic cloud. Are those 2 options enabled automatically as below?

Yes, node.attr.data is set to hot or warm as appropriate on the hot and warm nodes in an Elastic Cloud hot/warm deployment. You'll have to set up the policies with allocate actions yourself, though, but you seem to have an understanding of that as the policy you posted above includes that.

I didn't find "is_write_index": true is added to aliases block of the index template.
Do you know why?

Yes - this is one of the most common questions we have about ILM and it's probably the hardest part of the whole thing.

Only one index can be the write index for an alias at a time, although an alias can point to many indices as long as only one is the write index. When an index is rolled over (either via the rollover API or ILM), the new index is created, and then once it's ready, the alias' write index is changed to point at the new index. This makes sure that there's no gap during which writes to the alias will fail.

However, it means that the new index has to exist for a short time while the old index is still the write index for the alias. If you put is_write_index: true in the index template, when the new index is created, it will conflict with the old index, as the old index is still marked as the write index for the alias. Having the alias in the template is OK, since an alias can point to many indices, the template just can't have is_write_index: true set, as that has to be set as part of the roll over.

all in all, Our final aim make ILP automated without manual intervention after we defined the good policies.

Excellent! That's our goal with ILM - you set it up once, and then it simply works automatically from then on.

system · May 22, 2019, 5:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Metricbeat index lifecycle policy (ILM) not rolling over Elasticsearch ilm-index-lifecycle-management	3	2015	December 19, 2019
LifeCycle Policy Configuration Elasticsearch	38	1366	November 21, 2019
Lifecycle Policies current action blank Elasticsearch ilm-index-lifecycle-management	2	420	May 11, 2020
Index Life cycle Policy indexes not roll overing Elasticsearch	13	481	March 2, 2021
Rollover Doesn't work Elasticsearch	1	477	September 20, 2019

Elastic Cloud Kibana Index Lifecycle Policies is not working

Related topics