Rollover Alias with date?

[djtecha]

Hello All,
I'm having difficulty figuring out the correct API call to rollover an index to an hourly index name. Basically I need to rollover every hour if the index is over a certain size, and I have it working with a running counter but the format is suboptimal.

#ALIAS

  "test-2018.04.02-000001": {
    "aliases": {
      "test": {}
    }
  }
}

#Rollover

POST /grab/_rollover/%3Ctest-%7Bnow%2Fd%7D-%3E
{
  "conditions": {
    "max_age":   "1d",
    "max_size":  "500gb"
  }
}

Basically I can't figure out the proper URI encoded variation to make the new index be called test-2018.04.02-12 where the 12 here is the hour of the day.

[Christian_Dahlqvist]

Why do you need the date and time in the underlying index name? Kibana and Curator do no longer require this.

[djtecha]

It makes it easy to understand when an index was created. I am a fan of the date suffix but need to chop up daily indexes using rollover due to sizing constraints. Are you implying that when rolling over we should simply use an ever increasing counter suffix? What happens when I hit 999999?

[Christian_Dahlqvist]

Yes, that is what I am suggesting. If you ingest a lot of data and end up rolling over every hour, that range will still last you over 100 years, so I would not necessarily be too concerned about that.

[djtecha]

How can I be confident that when a DELETE occurs it's actually hitting an index older then say 2 weeks in this scenario? I suppose curator now knows the age of an index (which is cool) but adhoc deletes will have to be more careful.

[theuntergeek (Aaron Mildenstein)]

You could still create this with a dated index name, it just has to have at least a -1 at the end.

Check out the date math documentation:

Expressions can be pretty ornate, complete with formatting:

<logstash-{now/M-1M{YYYY.MM}}>

It sounds like what you'd need is more like:

<test-{now/d{YYYY.MM.dd.HH}}-1>

Which would create an index test-2018.04.05.16-1. Obviously you'll have to URL encode/escape this, but it will work.

Elasticsearch Curator has an undocumented (I'll get around to that eventually) setting to allow you to specify a new_index name for the Rollover action. You can use date math, because it just passes the value along to Elasticsearch, and does the URL encoding for you. Repeat: You don't have to URL encode this.

[theuntergeek (Aaron Mildenstein)]

Curator has 3 ways to determine index age: index name, creation_date, and field_stats. Field stats may be the most useful for you, as it actually queries the minimum or maximum age of a timestamp field in the index. With two filters, you could filter for both criteria. Because of this, you do not need to name an index with a date stamp in it.

Note: Don't let the name field_stats fool you, since the actual field_stats API was deprecated in 5.x. Curator preserved the name and just queries the fields now.

[Christian_Dahlqvist]

You can get the index creation timestamp through the cat indices API, e.g.: http://localhost:9200/_cat/indices?h=h,s,i,id,creation.date.string

[djtecha]

ok, I'll work on deprecating the Date suffix and tooling then. Thanks.