# Logstash - Elasticsearch 7.4 ILM

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [March 12, 2020, 5:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389 "2020-03-12T17:52:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fvegaucr1](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@fvegaucr1](https://discuss.elastic.co/u/fvegaucr1)\
**Post date:** [March 12, 2020, 5:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/1 "2020-03-12T17:52:07Z")

</div>

Hi Im trying to set up ILM for my indices. But just want to confirm that what Im doing is ok.

First we have 4 locations in differents parts of the world. We use filebeat to get the logs and send it to a kafka cluster under a specific topic name to identify the region like index1\_ams1. The logstash will pull the information perfrom the filtering and create the index in centralized elasticsearch cluster:

The problem that I have not sure if I do, is that how do I know I set ILM correctly.

So this is what I did:  
a- Create a policy [http://app1417.atl1.XXX.com:9200/\_ilm/policy](http://app1417.atl1.XXX.com:9200/_ilm/policy)

> ```
> "xxx_policy" : {
> "version" : 1,
> "modified_date" : "2020-03-11T22:47:02.124Z",
> "policy" : {
> "phases" : {
> "warm" : {
> "min_age" : "30d",
> "actions" : {
> "allocate" : {
> "number_of_replicas" : 2,
> "include" : { },
> "exclude" : { },
> "require" : {
> "box_type" : "warm"
> }
> },
> "forcemerge" : {
> "max_num_segments" : 1
> },
> "set_priority" : {
> "priority" : 50
> },
> "shrink" : {
> "number_of_shards" : 1
> }
> }
> },
> "cold" : {
> "min_age" : "90d",
> "actions" : {
> "allocate" : {
> "number_of_replicas" : 1,
> "include" : { },
> "exclude" : { },
> "require" : {
> "box_type" : "cold"
> }
> },
> "set_priority" : {
> "priority" : 0
> }
> }
> },
> "hot" : {
> "min_age" : "0ms",
> "actions" : {
> "rollover" : {
> "max_size" : "50gb",
> "max_age" : "7d"
> },
> "set_priority" : {
> "priority" : 100
> }
> }
> },
> "delete" : {
> "min_age" : "91d",
> "actions" : {
> "delete" : { }
> }
> }
> }
> }
> }
> 
> ```

b- Then I create a template for the policy:

> ```
> "xxx_per_index_atl1" : {
> "order" : 0,
> "index_patterns" : [
> "xxx_atl1-*"
> ],
> "settings" : {
> "index" : {
> "lifecycle" : {
> "name" : "xxx_policy",
> "rollover_alias" : "xxx_atl1_ilm"
> },
> "routing" : {
> "allocation" : {
> "require" : {
> "node_type" : "hot"
> }
> }
> },
> "refresh_interval" : "5s",
> "analysis" : {
> "analyzer" : {
> "default" : {
> "type" : "standard",
> "stopwords" : "_none_"
> }
> }
> },
> "number_of_shards" : "45"
> }
> },
> "mappings" : {
> "dynamic_templates" : [
> {
> "javathread" : {
> "mapping" : {
> "norms" : false,
> "index" : true,
> "type" : "text",
> "index_options" : "offsets"
> },
> "match_mapping_type" : "string",
> "match" : "javathread"
> }
> },
> {
> "message" : {
> "mapping" : {
> "norms" : false,
> "index" : true,
> "type" : " text",
> "index_options" : "offsets"
> },
> "match_mapping_type" : "string",
> "match" : "message"
> }
> },
> {
> "stacktrace" : {
> "mapping" : {
> "norms" : false,
> "index" : true,
> "type" : "text",
> "index_options" : "offsets"
> },
> "match_mapping_type" : "string",
> "match" : "stacktrace"
> }
> },
> {
> "string_fields" : {
> "mapping" : {
> "type" : "multi_field",
> "fields" : {
> "{name}" : {
> "norms" : false,
> "index" : true,
> "type" : "keyword",
> "index_options" : "offsets"
> }
> }
> },
> "match_mapping_type" : "string",
> "match" : "*"
> }
> }
> ],
> "properties" : {
> "severity" : {
> "index" : true,
> "type" : "keyword"
> },
> "geoip" : {
> "dynamic" : true,
> "type" : "object",
> "properties" : {
> "ip" : {
> "type" : "ip"
> },
> "location" : {
> "type" : "geo_point"
> }
> }
> },
> "syslog_severity_code" : {
> "index" : true,
> "type" : "keyword"
> },
> "@version" : {
> "index" : true,
> "type" : "keyword"
> },
> "syslog_facility_code" : {
> "index" : true,
> "type" : "keyword"
> },
> "pid" : {
> "index" : true,
> "type" : "keyword"
> },
> "priority" : {
> "index" : true,
> "type" : "keyword"
> },
> "facility" : {
> "index" : true,
> "type" : "keyword"
> },
> "tags" : {
> "index" : true,
> "type" : "keyword"
> }
> }
> },
> "aliases" : { }
> },
> ".ml-state" : {
> "order" : 0,
> "version" : 7040299,
> "index_patterns" : [
> ".ml-state*"
> ],
> "settings" : {
> "index" : {
> "auto_expand_replicas" : "0-1",
> "unassigned" : {
> "node_left" : {
> "delayed_timeout" : "1m"
> }
> }
> }
> },
> "mappings" : {
> "_meta" : {
> "version" : "7.4.2"
> },
> "enabled" : false
> },
> "aliases" : { }
> }
> }
> 
> ```

Im using logstash to create the index here is the config:

> ```
> if [type] == "xxx" and [index_name] {
> elasticsearch {
> index => "xxx_atl1-%{+YYYY.MM.dd}"
> hosts => ["app1417.atl1.xxx.com:9200","app1431.atl1.xxx.com:9200"]
> manage_template => false
> template_name => "xxx_per_index_atl1"
> template => "/etc/logstash/templates/xxx.json"
> template_overwrite => true
> ilm_enabled => true
> ilm_rollover_alias => "xxx_atl1_ilm"
> ilm_policy => "xxx_policy"
> }
> }
> }...
> 
> ```

So far that is ok, then I see that Elasticsearch created this index:

> ```
> xxx_atl1_ilm-2020.03.11-000001
> 
> ```

But when I check ilm explain I get this [http://app1417.atl1.xxx.com:9200/xxx\_ams1\*/\_ilm/explain](http://app1417.atl1.xxx.com:9200/xxx_ams1*/_ilm/explain)

> ```
> {
> "indices" : {
> "xxx_atl1-2020.03.10" : {
> "index" : "xxx_atl1-2020.03.10",
> "managed" : false
> },
> "xxx_atl11-2020.03.07" : {
> "index" : "xxx_atl1-2020.03.07",
> "managed" : false
> },
> "xxx_atl1-2020.03.08" : {
> "index" : "xxx_atl1-2020.03.08",
> "managed" : false
> },
> "xxx_atl1-2020.03.11" : {
> "index" : "xxx_atl1-2020.03.11",
> "managed" : false
> },
> "xxx_atl1-2020.03.09" : {
> "index" : "xxx_atl1-2020.03.09",
> "managed" : false
> },
> "xxx_atl1_ilm-2020.03.11-000001" : {
> "index" : "xxx_atl1_ilm-2020.03.11-000001",
> "managed" : false
> }
> }
> }
> 
> ```

and the ilm status is running:  
[http://app1417.atl1.xxx.com:9200/\_ilm/status](http://app1417.atl1.xxx.com:9200/_ilm/status)

> ```
> {
> "operation_mode" : "RUNNING"
> }
> 
> ```

Also [http://app1417.atl1.xxx.com:9200/\_cluster/settings](http://app1417.atl1.xxx.com:9200/_cluster/settings) show

> ```
> {
> "persistent" : { },
> "transient" : { }
> }
> 
> ```

When I look at the index created I see:

> ```
> {
> "xxx_atl1_ilm-2020.03.11-000001" : {
> "aliases" : {
> "xxx_atl1_ilm" : {
> "is_write_index" : true
> }
> },
> ...
> "settings" : {
> "index" : {
> "creation_date" : "1583968214030",
> "number_of_shards" : "1",
> "number_of_replicas" : "1",
> "uuid" : "mwsJqfdjSFeIW5YaTcLCyQ",
> "version" : {
> "created" : "7040299"
> },
> "provided_name" : "<xxx_atl1_ilm-{now/d}-000001>"
> }
> }
> }
> }
> 
> ```

Can someone let me know if this looks ok

---

<div class="post-metadata">

**Author:** ![fvegaucr1](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@fvegaucr1](https://discuss.elastic.co/u/fvegaucr1)\
**Post date:** [March 19, 2020, 4:16pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/2 "2020-03-19T16:16:48Z")

</div>

Can someone help with this please?

---

<div class="post-metadata">

**Author:** ![fvegaucr1](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@fvegaucr1](https://discuss.elastic.co/u/fvegaucr1)\
**Post date:** [April 9, 2020, 8:16pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/3 "2020-04-09T20:16:37Z")

</div>

@Badger any idea?

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [May 1, 2020, 6:29pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/4 "2020-05-01T18:29:20Z")

</div>

This can be closed.  
I manage to fix, the issue was the way I set up the patterns and the aliases.  
I got it working.

Thank you all

---

<div class="post-metadata">

**Author:** ![elephant-stone](https://avatars.discourse-cdn.com/v4/letter/e/a9a28c/32.png) [@elephant-stone](https://discuss.elastic.co/u/elephant-stone)\
**Post date:** [May 7, 2020, 8:22pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/5 "2020-05-07T20:22:21Z")

</div>

@fvegaucr could you please share what was wrong with regards to configuration in the first post? I'm struggling to got it working. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 4, 2020, 8:22pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-7-4-ilm/223389/6 "2020-06-04T20:22:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.