Elasticsearch custom ILM

[darrylrmiles (Darryl Miles)]

Hello,

I am attempting to setup custom indexes for winlogbeat via logstash using ILM settings in the elasticsearch output. I would like to use the default template for winlogbeat (winlogbeat-7.3.0) but have different custom indexes for each organization. The purpose of the separate indexes is to have users be able to login to the kibana instance with roles that can only view their particular indexes for winlogbeat. I would like to use the following base structure for the indexes

winlogbeat-{agent.version}-organization

I figured appending the organization to the end would allow me to use the default template for winlogbeat according to each version as well as to use the built in dashboards and visualizations for winlogbeat-*. Also, it would allow me to use a default ILM policy for all organization indexes just by specifying the index pattern in a specific ILM template similar to how it is described here

Implementing Hot-Warm-Cold in Elasticsearch with Index Lifecycle Management

My problem is mainly when specifying the output for ilm_rollover_alias in logstash's elasticsearch output it is not appending the date and increment number like I would hope. Here is the logstash config:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "elastic_cloud_url"
    user => "user"
    password => "password"
    ilm_enabled => true
    ilm_rollover_alias => "%{[@metadata][beat]}-%{[@metadata][version]}-miles"
    ilm_policy => "hot-warm-delete-30days"
  }
}

So it is creating the index and applying the correct ILM policy but the index name is just what is in the rollover alias without the ilm_pattern appended to the end:

winlogbeat-7.3.0-miles

It also seems to be using the correct winlogbeat-7.3.0 template from what I can tell on the index settings since I configured that one in particular to have 5 shards and 1 replica. Below is snippet pulled from index settings via kibana:

{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "hot-warm-delete-30days",
        "rollover_alias": "winlogbeat-7.3.0"
      },
      "routing": {
        "allocation": {
          "require": {
            "data": "hot"
          }
        }
      },
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "5",
      "provided_name": "winlogbeat-7.3.0-miles"

I noticed in the snippet above also that the rollover_alias is the default from the winlogbeat-7.3.0 template so something doesn't seem correct. I am getting the error in kibana

illegal_argument_exception: index.lifecycle.rollover_alias [winlogbeat-7.3.0] does not point to index [winlogbeat-7.3.0-miles]

Am I able to structure my data and policies in this way or am I way off base here?

[rugenl (Len Rugen)]

You can do it, it's similar to what I'm doing.... getting there is all the fun.

Did you run winlogbeat setup from winlogbeat as configured? I think you need a template for each index name pattern. I think running setup for each config would create a more-qualified index name.

Here's how I did our setup, there are any ways...

1. I created the default beats template
2. I created a higher priority template for each unique index, but it is small, just adds ILM settings to the new index.
3. In beats elasticsearch output, I used index name instead of the ilm_ values. That let me use variables where you have "miles".
4. I created a "boot-strap" index each time, before starting beats for that index.

[darrylrmiles (Darryl Miles)]

I think you set me on the right track. I ended up finding this article regarding apm manual index lifecycle management

https://www.elastic.co/guide/en/apm/server/current/manual-ilm-setup.html

I did load the default beats template originally so it has order of 1. I ended up adding another template specifically for my custom index

PUT _template/winlogbeat-7.3.0-miles-ilm
{
"order": 2,
"index_patterns": ["winlogbeat-7.3.0-miles*"],
"settings": {
"index.lifecycle.rollover_alias": "winlogbeat-7.3.0-miles",
"index.lifecycle.name": "hot-warm-delete-30days"
}
}

I then manually added the index with the alias association (don't believe I need date in index for this to work)

PUT winlogbeat-7.3.0-miles-000001
{
"aliases": {
"winlogbeat-7.3.0-miles": {
"is_write_index": true
}
}
}

So now logstash elasticsearch output is still using the ilm_rollover_alias with metadata to form the value winlogbeat-7.3.0-miles for this beat and is writing to the newly created index.

  "settings": {
    "index": {
      "lifecycle": {
        "name": "hot-warm-delete-30days",
        "rollover_alias": "winlogbeat-7.3.0-miles"
      },
      "routing": {
        "allocation": {
          "require": {
            "data": "hot"
          }
        }
      },
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "5",
      "provided_name": "winlogbeat-7.3.0-miles-000001"

Seems like this will work but I feel like i'm missing something that should make this a bit easier and more dynamic than the manual method

[rugenl (Len Rugen)]

I had a somewhat related case with Elastic. Paraphrased, with ILM, there is setup, it's just the cost of ILM.

It seems from other posts here that missing the manual creation of the initial index is often missed.

Having a date in the index is a tradeoff, and to some extent, adds confusion. It is the date the index is created, if it takes many days to rollover, it will of course contain other events from later dates. We are using dates since we started with the old style daily indices.

Finally, you will have to prepare the system before you let a future beats version. We are to try to only support selected clients, not every one, then encourage timely client migration to the new target version.

[darrylrmiles (Darryl Miles)]

Thanks for the explanation. I think that the time spent to do the up front setup is a good trade off to having better index management.

Thanks again for the direction and help!