How to attach Index Lifecycle Policy to indices created by Logstash?

[pritster5]

I have a Logstash Config file that looks like this:

input {
    s3 {
        bucket => "testbucket"
        region => "us-east-1"
        codec => "json"
        additional_settings => {
            force_path_style => true
            follow_redirects => false
        }
    }
}

output {
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "test.xxx.blahblahblah.output-%{+YYYY.MM.dd}"
        user => xxx
        password => yyy
    }
}

I want all indices created by Logstash (a new one is created daily) to obey an Index Lifecycle Policy I have created which looks like this:

PUT _ilm/policy/index-loop
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

How would I do this?

I currently do:

PUT test*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "index-loop"
    }
  }
}

which matches against all the indices created by logstash, but I don't know if this will apply to indices created AFTER this put request was run.

[leandrojmp (Leandro Pereira)]

Do you have a template for your index?

You will need to create a template for your index and add the lifecycle policy in it.

Something like this:

  "index": {
    "lifecycle": {
      "name": "policy-name"
    }

[pritster5]

I do not. How do I make an index template for an index created by logstash?

[leandrojmp (Leandro Pereira)]

You just need to create an index template in Elasticsearch and set the index_pattern of the template to match the names that you are using in logstash.

For example, using the index_pattern of test.* will match any index created which name starts with test.

The documentation about index templates is here

[pritster5]

If I'm making the index template through Kibana's GUI, would I put the

"index": {
    "lifecycle": {
      "name": "index-loop"
    }

inside here?:

And is the index policy itself stored internally after I execute?:

PUT _ilm/policy/index-loop
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

[pritster5]

SOLUTION (for ELK 7.5 at least):

1. Create an Index Template with an Index Pattern that matches the Index name defined in your Logstash pipeline config.

2. Make an Index Lifecycle Management Policy via Kibana GUI: Settings -> Elasticsearch -> Index Lifecycle Policies: Create Policy. Configure however you want.

3. Go back to the main Index Policy Menu, and click "Actions" next to the Index Policy you just made.

4. Click "Add policy to index template" and then select the index template we created in step 1.

5. Done. Any new indices created by Logstash that match the index pattern in your index template will automatically have the lifecycle policy we made applied to them.