# ILM Policy through logstash dynamic index name
**URL:** https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011
**Category:** Logstash
**Tags:** ilm-index-lifecycle-management
**Created:** [March 22, 2021, 7:56pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011 "2021-03-22T19:56:59Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [March 22, 2021, 7:56pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/1 "2021-03-22T19:56:59Z")
Hello Team,
I am trying to implement the ILM Policy but I ran into an issue because my index name is dynamically created through logstash. Let me give you an explain of my setup and wat I am trying to do
1. I create the template using the below API
2. Then I created the index
3. In my logstash setting I setup the below ilm policy settings
output {
elasticsearch {
```
ilm_rollover_alias => "logs"
ilm_pattern => "000001"
ilm_policy => "logs_policy"
```
}
}
All the above worked but now I want to create logs for each module so the logs that I am trying to send have a column **module\_name**. So in logstash I can setup index name dynamically and it will create **logs\_%{module\_name}** but if I do that I will have to setup ilm policy and templates for each of the modules right ?? Is there a better way to implement it?
New logstash for Dynamic index
filter {
json {
source =\> "message"
remove\_field =\> ["message", "@version"]
}
mutate{
lowercase =\> ["module\_name"]
}
}
output {
elasticsearch {
index =\> "logs\_%{module\_name}"
}}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 22, 2021, 10:16pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/2 "2021-03-22T22:16:33Z")
An elasticsearch output can support sprintf references for the index name because in the [bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) each indexing request includes the index name, so it can be copied from the event. However, ILM is applied when the output makes its initial connection to elasticsearch and at that point there is no event that can be referenced.
Do you want the same policy for all of the indexes? If so, you may be able to [apply the policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#apply-policy-multiple) after the index is created.
Note that having a large number of small indexes is a negative for performance in elasticsearch.
---
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [March 23, 2021, 2:00am UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/3 "2021-03-23T02:00:49Z")
thanks, @Badger for the quick response
Yes, I want the same policy to apply to all my indexes but when I use the **ilm\_rollover\_alias** in logstash as "logs" the rollover doesn't happen. In that case, what should be the alias settings?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 23, 2021, 3:22am UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/4 "2021-03-23T03:22:05Z")
I do not use elasticsearch, but I do not think the logstash elasticsearch output can support this. But why do you care? The rollover policy only matters when the index is rolled over. If it take 24 hours to do the rollover then if you have a cron job calls a curl statement to set the policy every 12 hours then the end result will be the same.
I realise that it feels much better to have the index configured better from the moment of its creation. I would prefer that too, but does it matter? You can beat ES to the punch (rollover) and set the policy before it takes effect.
---
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [March 31, 2021, 12:33pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/5 "2021-03-31T12:33:03Z")
Hello @Badger
I tried to implement the curl cmd using the below syntx,
curl -X PUT "localhost:9200/test-index?pretty" -H 'Content-Type: application/json' -d'
{
"settings": {
"number\_of\_shards": 1,
"number\_of\_replicas": 1,
"index.lifecycle.name": "my\_policy"
}
}
'
but I am facing issues when I try to implement the policy. As it is needed to provide the "index\_pattern" which is logs\* in my case and the ilm\_pattern is "000001" so when the logstash creates an index as log\_module1 it is not getting picked and the rollover doesn't happen.
Can you explain the step-by-step follow I feel I am doing something wrong there?
Thanks a lot in advance.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 31, 2021, 2:35pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/6 "2021-03-31T14:35:25Z")
Try using the [method](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#apply-policy-multiple) I linked to.
---
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [March 31, 2021, 11:10pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/7 "2021-03-31T23:10:46Z")
I tried the following steps and run into the below error
1. Created the template and set ilm policy for that template
PUT \_template/logs\_template
{
"index\_patterns": ["testlogs\*"],
"settings": {
"number\_of\_shards": 2,
"number\_of\_replicas": 0,
"index.lifecycle.name": "logs\_policy",
"index.lifecycle.rollover\_alias": "testlogs"
}
}
1. Edited the policy and set the doc count to like 10 to test the rollover.
2. Sent the data using logstash to the index. index name "testlogs\_dataseers".
3. Applied the policy to the index after it was created
PUT testlogs\_dataseers/\_settings
{
"index": {
"lifecycle": {
"name": "logs\_policy"
}
}
}
can I know what I am doing wrong here.
---
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [April 5, 2021, 9:46pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/8 "2021-04-05T21:46:59Z")
wait to hear back from the community.
---
### Author:  [@gbandasha](https://discuss.elastic.co/u/gbandasha)
#### Post date: [April 8, 2021, 7:51pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/9 "2021-04-08T19:51:13Z")
waiting to hear back from the community.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 13, 2021, 7:05pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/10 "2021-04-13T19:05:07Z")
First clean everything up.
Here is a sample that works
```
PUT _ilm/policy/license-data
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "30d"
}
}
}
}
}
}
PUT _index_template/license-data
{
"index_patterns": [
"license-data*"
],
"template": {
"settings": {
"number_of_shards": 1,
"lifecycle": {
"name": "license-data",
"rollover_alias": "license-data"
}
}
}
}
```
Logstash Conf :` simple-conf.conf`
```
##################################
# Read License file
##################################
input {
file {
path => "/Users/sbrown/workspace/elastic-install/7.11.2/logstash-7.11.2/LICENSE.txt"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
# pump to stdout for debug
stdout {codec => rubydebug}
elasticsearch {
hosts => ["localhost:9200"]
ilm_rollover_alias => "license-data"
ilm_pattern => "000001"
ilm_policy => "license-data"
}
}
```
Run logstash
`sudo ./bin/logstash -r -f ./simple-file.conf`
```
GET _cat/aliases/license-data?v
GET _cat/indices/license-data-000001
GET /license-data
```
Results
```
# GET _cat/aliases/license-data?v
alias index filter routing.index routing.search is_write_index
license-data license-data-000001 - - - true
# GET _cat/indices/license-data-000001
yellow open license-data-000001 ZhlSamExR1KqTNdCNQUjCw 1 1 223 0 93.2kb 93.2kb
# GET /license-data
{
"license-data-000001" : {
"aliases" : {
"license-data" : {
"is_write_index" : true
}
},
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"settings" : {
"index" : {
"lifecycle" : {
"name" : "license-data",
"rollover_alias" : "license-data"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "",
"creation_date" : "1618338466085",
"number_of_replicas" : "1",
"uuid" : "ZhlSamExR1KqTNdCNQUjCw",
"version" : {
"created" : "7110199"
}
}
}
}
}
```
With respect to dynamic index name you will need the policies and templates in place for all the combinations.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 11, 2021, 7:05pm UTC](https://discuss.elastic.co/t/ilm-policy-through-logstash-dynamic-index-name/268011/11 "2021-05-11T19:05:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.