How can I name Curator rollover new Index like date prefix-YYYY.MM.DD-1?

Elastic Stack Elasticsearch
1

Hi!!
I want curator rollover by date (prefix-YYYY.MM.DD-1) like description at:
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/rollover.html

My configuration is:

  • Create index:
PUT /logsservices-2018.04.17-1
  • Create Alias:
POST /_aliases
{
  "actions" : [
    { "add" : { "indices" : ["logsservices-2018.04.17-1"], "alias" : "logsservices" } }
  ]
}
  • Config action:
actions:
  1:
    action: rollover
    description: >-
      Rollover the index associated with alias 'aliasname', which should be in the
      form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
    options:
      name: logsservices
      conditions:
      max_age: 1d
      max_docs: 1000000
      extra_settings:
        index.number_of_shards: 5
        index.number_of_replicas: 1

However, at next day I get this on log:
//2018-04-18 12:00:17,880 DEBUG elasticsearch log_request_success:86 < {"old_index":"logsservices-2018.04.17-1","new_index":"logsservices-2018.04.17-000002","rolled_over":true,"dry_run":false,"acknowledged":true,"shards_acknowledged":true,"conditions":{"[max_docs: 1000000]":false,"[max_age: 1d]":true}}

and created a new index with name "logsservices-2018.04.17-000002" instead of "logsservices-2018.04.18-1"

what am I making wrong? How is the correct configuration?

Thanks!!!

2

I took the liberty of editing your post to make it legible. Please use the code tags or triple back-ticks to encapsulate preformatted text in the future.

First, you can create the index and the alias in one shot:

PUT /logservices-2018.04.17-1
{
  "aliases": {
    "logservices": {}
  }
}

Second, there is an undocumented feature for the rollover action that lets you name a new index:

  options:
    ...
    new_index: '<logservices-{now/d}-1}>'

If you need something other than the current year-month-day, look at the date math docs and follow the conventions there.

3

On a separate note, why even both with dated indices if you're using rollover? You can still use Curator's other age filter options (i.e. field stats) to calculate the index age without needing the index to have a timestamp in the index name. You're generally better off letting the index grow to 5+ GB per shard before rolling over anyway, as smaller than that is just wasted resources.

I heavily advocate just naming the indices logservices-000001 and rolling over to just the next number.

2 Likes
4

Great!! I like this solution but I have a question.

I'm thinking use delete_indices action for delete older logs and the Elasticsearch version is 6.2. Should I use field_caps or field_stats on filter age to get field @timestamp?

Thanks!!!

5

Curator still calls it field_stats, but it's actually just doing a query and min or max aggregations now. Just use that with Curator.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.