ILM Logstash and Elastic (again)

Elastic Stack Logstash
1

Hi,

I have following configurations:

logstash output:

output {
  elasticsearch {
    hosts => ["elastic.server.com"]
	ilm_rollover_alias => "filebeat-vm-linux"
    ilm_pattern => "{now/d}-000001"
	ilm_policy => "test_policy"
	ilm_enabled => true
	ssl => true
	ssl_certificate_verification => true
	cacert => 'ca.crt'
    user => logstash   
    password => "${LOGSTASH}"
  }

test_policy:

PUT _ilm/policy/test_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1m",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "1m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

index template:

PUT _template/linux_vm_template
{
  "order": 0,
  "index_patterns": [
    "filebeat-vm-linux*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "test_policy",
        "rollover_alias": "filebeat-vm-linux"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "_doc": {
      "_meta": {},
      "_source": {},
      "properties": {}
    }
  }
}

Expected result:

First index:
filebeat-vm-linux-{now/d}-000001

every minute (just for testing) new index is getting created as defined in the policy:

filebeat-vm-linux-{now/d}-000002
filebeat-vm-linux-{now/d}-000003
filebeat-vm-linux-{now/d}-000004
filebeat-vm-linux-{now/d}-000005

and so on.

Observated result:

filebeat-vm-linux was created and is constantly growing without any rollover actions.

In addition to that I have an error appearing in the index management in kibana:

illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-vm-linux] does not point to index [filebeat-vm-linux]

Can anyone help me to understand why it does happen?

Thanks

2

I have modified the logstash output and added the template configuration as follows:

output {
  elasticsearch {
    hosts => ["elastic.server.com"]
    template_name => "linux_vm_template"
    template_overwrite => true
	ilm_rollover_alias => "filebeat-vm-linux"
    ilm_pattern => "{now/d}-000001"
	ilm_policy => "test_policy"
	ilm_enabled => true
	ssl => true
	ssl_certificate_verification => true
	cacert => 'ca.crt'
    user => logstash   
    password => "${LOGSTASH}"
  }

after that I have removed the lifecycle part from the index template (since the logstash output now will take care of updating the template with the right information as far as I understood).

Now the index filebeat-vm-linux-{now/d}-000001 is created correctly and I have no errors anymore, but still the index is not getting rolled over as stated in the policy. Now I have a growing filebeat-vm-linux-{now/d}-000001 index

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.