# ILM settings not being applied **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [August 4, 2020, 3:39am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643 "2020-08-04T03:39:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![wardallen](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@wardallen](https://discuss.elastic.co/u/wardallen)\ **Post date:** [August 4, 2020, 3:39am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/1 "2020-08-04T03:39:20Z")
I have a logstash index storing data that will be used in reporting, but only ever for the last 5 minutes. Hence, I want to have a very aggressive deletion policy. Roll the index every 15 minutes Delete the rolled (old) index. However, my ILM, index template and index creation must have a flaw in it, because every time I view the ILM status of my index, my settings aren't there. Can someone please point out where I'm going wrong? ILM POLICY ```auto PUT /_ilm/policy/logstash-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "15m", "max_size": "10GB" } } }, "delete": { "min_age": "15m", "actions": { "delete": {} } } } } } ``` INDEX TEMPLATE ```auto PUT /_template/logstash_template { "index_patterns": ["logstash-*"], "settings": { "number_of_shards": 1, "number_of_replicas": 0, "index.lifecycle.name": "logstash-policy", "index.lifecycle.rollover_alias": "logstash-alias" } } ``` INDEX CREATION ```auto PUT /logstash-000001 { "settings": { "number_of_shards": 1, "number_of_replicas": 0 }, "aliases": { "logstash":{ "is_write_index": true } }, "mappings": { } } ``` And when I query the ILM settings, I don't see the things I expect. ```auto GET logstash*/_ilm/explain?pretty { "indices" : { "logstash-000001" : { "index" : "logstash-000001", "managed" : true, "policy" : "logstash-policy", "lifecycle_date_millis" : 1596507802885, "age" : "1.16h", "phase" : "hot", "phase_time_millis" : 1596507802959, "action" : "rollover", "action_time_millis" : 1596508792079, "step" : "check-rollover-ready", "step_time_millis" : 1596508792079, "phase_execution" : { "policy" : "logstash-policy", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "50gb", "max_age" : "30d" } } }, "version" : 1, "modified_date_in_millis" : 1595979741503 } } } } ``` The max\_size and max\_age values look like they came from a default somewhere. Can someone spot where I'm going wrong? ---
**Author:** ![wardallen](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@wardallen](https://discuss.elastic.co/u/wardallen)\ **Post date:** [August 6, 2020, 3:55am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/2 "2020-08-06T03:55:15Z")
I think I found it, the problem seems to have been in the index creation code. I should have put aliases.logstash-alias.is\_write\_index instead of aliases.logstash.is\_write\_index ```auto INDEX CREATION PUT /logstash-000001 { "settings": { "number_of_shards": 1, "number_of_replicas": 0 }, "aliases": { "logstash-alias":{ "is_write_index": true } }, "mappings": { } } ``` However, that's brought a new issue to light. In the field mappings, I define a field as type=ip so I can do CIDR searches. Something, I suspect logstash, is changing that back to a type=text. Here's my logstash config ```auto input { tcp { port => 10514 type => bc_syslog } } filter { # drop comment lines if ([message] =~ /^#/) { drop{} } csv { columns => ["date", "time", "time-taken", "c-ip", "cs-username", "cs-auth-group", "s-supplier-name", "s-supplier-ip", "s-supplier-country", "s-supplier-failures", "x-exception-id", "sc-filter-result", "cs-categories", "cs-Referer", "sc-status", "s-action", "cs-method", "rs-Content-Type", "cs-uri-scheme", "cs-host", "cs-uri-port", "cs-uri-path", "cs-uri-query", "cs-uri-extension", "cs-User-Agent", "s-ip", "sc-bytes", "cs-bytes", "x-virus-id", "x-bluecoat-application-name", "x-bluecoat-application-operation", "r-ip", "cs-threat-risk", "x-bluecoat-transaction-uuid", "x-icap-reqmod-header-X-ICAP-Metadata", "x-icap-respmod-header-X-ICAP-Metadata", "x-rs-certificate-validate-status", "x-rs-certificate-observed-errors", "x-cs-ocsp-error", "x-rs-ocsp-error", "x-rs-connection-negotiated-ssl-version", "x-rs-connection-negotiated-cipher", "x-rs-connection-negotiated-cipher-size", "x-rs-certificate-hostname", "x-rs-certificate-hostname-category", "x-cs-connection-negotiated-ssl-version", "x-cs-connection-negotiated-cipher", "x-cs-connection-negotiated-cipher-size", "x-cs-certificate-subject", "s-sitename", "x-rs-certificate-hostname-threat-risk", "rs-bytes", "c-starttime", "filelength", "filesize", "avgbandwidth", "x-rs-streaming-content", "x-streaming-rtmp-app-name", "x-streaming-rtmp-stream-name", "x-streaming-rtmp-swf-url", "x-streaming-rtmp-page-url", "s-dns", "s-session-id", "x-cache-info", "s-port"] separator => " " } # parse timestamp if [date] and [time] { mutate { add_field => { "timestamp" => "%{date} %{time}" } } date { match => ["timestamp", "YYYY-MM-dd HH:mm:ss"] timezone => ['Australia/Queensland'] } } } output { elasticsearch { hosts => ["localhost:9200"] ilm_rollover_alias => "logstash-alias" ilm_pattern => "000001" ilm_policy => "logstash-policy" } } ``` Am I missing something in logstash to tell it to leave the index fields alone? ---
**Author:** ![Stuart\_Duncan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stuart_duncan/32/73423_2.png) [@Stuart\_Duncan](https://discuss.elastic.co/u/Stuart_Duncan)\ **Post date:** [August 6, 2020, 6:45am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/3 "2020-08-06T06:45:52Z")
I'd recommend you raise a new issue for this, since it looks like you have resolved the ilm issue and this is now about field mappings. That said, have you considered using the Ingest Pipeline functionality in Elasticsearch to do the CSV extraction, rather than Logstash? [https://www.elastic.co/guide/en/elasticsearch/reference/master/csv-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/csv-processor.html) And [https://www.elastic.co/blog/indexing-csv-elasticsearch-ingest-node](https://www.elastic.co/blog/indexing-csv-elasticsearch-ingest-node) This would give you more control at Elasticsearch, and remove some dependency on Logstash. If you are still having issues, in a new Topic please share: a) You're mapping template, and b) an example raw event And we can delve a bit deeper! Stuart. ---
**Author:** ![wardallen](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@wardallen](https://discuss.elastic.co/u/wardallen)\ **Post date:** [August 6, 2020, 6:54am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/4 "2020-08-06T06:54:05Z")
Cheers Stuart, but again I think I found what I was doing wrong. I didn't realise logstash was creating the indexes (according to the template), so my field definitions should've gone in the template, not the index. ---
**Author:** ![Stuart\_Duncan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stuart_duncan/32/73423_2.png) [@Stuart\_Duncan](https://discuss.elastic.co/u/Stuart_Duncan)\ **Post date:** [August 6, 2020, 6:55am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/5 "2020-08-06T06:55:36Z")
Great! Glad you got it fixed - yes, the index template is where these need to be defined! Pleased its working for you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2020, 6:55am UTC](https://discuss.elastic.co/t/ilm-settings-not-being-applied/243643/6 "2020-09-03T06:55:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.