ILM setup: Logstash vs elasticsearch

[fefo69 (Federico)]

Dear all,

I'm trying to create indexes from logstash output plugin but policy assignment and rollover is not working.
Elasticsearch and logstash version 7.2

This is my test setup:

Logstash output plugin:

elasticsearch {
  hosts => [ "<ip>:9200" ]
  ilm_enabled => true
  ilm_rollover_alias => "test_roll_alias"
  ilm_pattern => "000001"
  ilm_policy => "test_policy"
}

Policy:

"test_policy": {
"version": 4,
"modified_date": "2019-07-26T14:23:35.741Z",
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50mb"
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}

Template

"test_template": {
"order": 0,
"index_patterns": [
"tef_switch*"
],
"settings": {
"index": {
"lifecycle": {
"name": "test_policy",
"rollover_alias": "test_roll_alias"
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {},
"aliases": {}
}

When I start getting data, index is created by logstash, but not linked to the policy and not doing the rollover when exceed the 50mb.

Appreciate your help to understand the setup of logstash output vs template/policy definitions in elasticsearch cluster (7.2/basic license)

Thanks!

[rugenl (Len Rugen)]

Did you create a bootstrap index? See https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management

There isn't a link to that section, search for "bootstrap"

[fefo69 (Federico)]

Hi

How can I handle bootsrap from logstash output plugin ?

Thanks !

[rugenl (Len Rugen)]

The doc always shows bootstrap via curl or the kibana dev tools console.

It reads like "filebeat setup --index-management" would also do it, but I've never had it work correctly, but I'm still 6.x.