Create Index lifecycle policy in Logstash output to Elasticsearch [step by step]

I have read almost all the topics on the Elastic page (tutorials) and topics from this forum, but I still have doubts how to set these settings.

The policy on the life cycle of the index and Rolling over.

Logstash config:

input {
  beats {
    port => 5044
}
filter {
  if "tracking" in [tags] {
grok {
  patterns_dir => ["/opt/logstash/patterns"]
  match => {"message" => "%{MEMORY}"}
  add_tag => ["memory"]
}
grok {
  patterns_dir => ["/opt/logstash/patterns"]
  match => {"message" => "%{CPU}"}
  add_tag => [ "cpu"]
}
grok {
  patterns_dir => ["/opt/logstash/patterns"]
  match => {"message" => "%{NUMBER:app_time}%{COMA}%{APP:app_type}%{COMA}%{MSG:app_msg}%{END}"}
}
  } else if "serial_log" in [tags] {
grok {
  match => { "source" => "%{IP:[host][name]}"}
}
  }
  mutate {
remove_tag => ["_grokparsefailure", "beats_input_codec_plain_applied"]
  }
}
}
output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

each index in above config is created each day with new date i.e : filebeat-2019.04.24 , filebeat-2019.04.25 and so on. But with time I have these indexes over 30 (e.g. after a month) and I would like them to be automatically removed e.g. after 14 days or size more than e.g. 20gb. That's why I have tried to use Index lifecycle policy. But still I cannot figure it out how this feature works

I would like to create based on my config output to ILM based on example from elastic site:

output {
  elasticsearch {
    ilm_rollover_alias: "filbeat"
    ilm_pattern: 
    ilm_policy: "test_policy"
  }
}
  1. Should I set: ilm_enabled => true
  2. Should I set above parameters i.e: ilm_rollover_alias: "custom" or rather ilm_rollover_alias => "custom"
  3. Sould I kept my index name as it is now: index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" , which give index named i.e: filebeat-2019.04.25 or point index to something else ?
  4. Should I set any index alias ?
  5. What I should set in ilm_pattern
  6. Should I use any index template ?

From Elastic/Kibana point of view i did:

Create policy:

{
  "test_policy" : {
    "version" : 1,
    "modified_date" : "2019-04-25T11:59:26.843Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "14d"
            }
          }
        },
        "delete" : {
          "min_age" : "2d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Create Template:

{
  "test_policy" : {
    "order" : 10,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "test_policy"
        },
        "routing" : {
          "allocation" : {
            "require" : {
              "data" : "hot"
            }
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}
  1. Should I add any other options or point to something ?

I did a tutorial from: tutorial_elastic_index_lifecycle
but with no effect.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.