Logstash ILM rollover alias unexpectedly treated as Data Stream after backing index deletion

Elastic Stack Elasticsearch
1

Hello Folks,

I am using Elastic Cloud v8.16. I am sending some data to Elasticsearch using Logstash.Below is the logstash configuration (output to elastic)

elasticsearch {
hosts => '${ELASTIC_URL}'
user => "${ELASTIC_USERNAME}"
password => "${ELASTIC_PASSWORD}"
ilm_rollover_alias => "logs-nginx-prod"
ilm_pattern => "000001"
ilm_policy => "logs-15-days"
manage_template => false
codec => json_lines
}
}

For this nginx i have created template.
PUT _index_template/sailpoint_logs
{
"priority": 1000,
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "logs-15-days",
"rollover_alias": "logs-nginx-prod"
},
"number_of_replicas": "1"
}
}
},
"index_patterns": [
"logs-nginx-*"
],
"composed_of": [
"nginx@mappings"
],
"ignore_missing_component_templates": [],
"allow_auto_create": true,
"_meta": {
"description": "index template for logstash-output-elasticsearch",
"author": "Aniket Pant"
}
}

Earlier i.e 15 days before , Index was created named logs-nginx-prod-000001 . After 15 days index got deleted and logstash wants to create the index but it couldn't create the index , so i check the logs and i found something . Please check the below logs.

:response=>{"index"=>{"_index"=>"logs-nginx-prod", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams", "caused_by"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [logs-nginx-prod]", "resource.type"=>"index_or_alias", "excluded_ds"=>"true", "resource.id"=>"logs-nginx-prod", "index_uuid"=>"_na_", "index"=>"logs-nginx-prod"}}}}}

I come to know that it want to create data stream based index and i haven't mentioned anything related to data stream , not even in template. I tried to add action = > index on logstash config and restarts the logstash again i am seeing the same error. I checked on kibana(Index management) logs-nginx-prod data stream index present with no documents and logs-nginx-prod template(default mapping and no ilm config).

I deleted the data stream index(log-nginx-prod) and updated the logstash config mentioned below.

elasticsearch {
hosts => '${ELASTIC_URL}'
user => "${ELASTIC_USERNAME}"
password => "${ELASTIC_PASSWORD}"
ilm_rollover_alias => "logs-nginx-prod"
ilm_pattern => "{now/d}-000001"
ilm_policy => "logs-15-days"
manage_template => false
codec => json_lines
index => "logs-nginx-prod"
}
}

Index got created logs-nginx-prod-2026.08.07-000001 . I don't know how data stream index got created when i am not defining anything . Is just because of my index is starting with logs-* and it will create data stream automatically .

2

Anything starting with logs-* will match some built-in templates, that will create data streams, you can still use a custom template matching your logs-nginx-prod index, but you need to set a priority higher than 200.

Another option, which is what I use and find easier, is to use a different name pattern that will not start with logs-*, I use a different prefix, something like data-* for example.

3

Hello @leandrojmp ,

I agree with you . As i mentioned that i have set the priority to 1000 but still it created data stream .