Curator rollover new_index failing

Curator action file:

actions:
  1:
    action: rollover
    options:
      disable_action: False
      name: index_prod_active
      new_index: "<index_prod_v1_{now/d{YYYY_MM_dd}}-1>" # have also tried "<index_prod_v1_{now/d{YYYY_MM_dd}}-000001>"
      conditions:
        max_age: 1d
        max_size: 8gb #this value is for testing purposes only

Message from logs:

ERROR     Failed to complete action: rollover.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(400, 'resource_already_exists_exception', 'index [index_prod_v1_2018_09_27-1/A7OAj2mZR2u0YHuGyyzKVw] already exists')

Expected behavior:

  1. new index created with name as such when conditions are met:
index_prod_v1_2018_09_27-2

I have followed the documentation here
I have read through this forum post
ES version - 6.2.3
Curator version - 5.5.4
I need to keep indices arranged by date, but I am also trying to keep shard sizes down to around ~20GB when all is said and done.

The problem you are encountering is because you're manually specifying the target index, and it's the same timestamp as the current day. Besides which, when you manually specify the -1 at the end, it can't auto-increment.

Allow me to demonstrate. I made this index down to the second to show the date rollover, using <index_prod_v1_{now/s{YYYY_MM_dd_HH_mm_ss}}-1>. You must url encode the date_math when using Console (but not in Curator—it handles it for you).

PUT /%3Cindex_prod_v1_%7Bnow%2Fs%7BYYYY_MM_dd_HH_mm_ss%7D%7D-1%3E
{
  "aliases": {
    "index_prod_active": {}
  }
}

yields:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "index_prod_v1_2018_09_27_15_55_33-1"
}

(Note: UTC time)

Subsequently, if I run:

POST /index_prod_active/_rollover 
{
  "conditions": {
    "max_age":   "1s"
  }
}

I see:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": "index_prod_v1_2018_09_27_15_55_33-1",
  "new_index": "index_prod_v1_2018_09_27_15_57_04-000002",
  "rolled_over": true,
  "dry_run": false,
  "conditions": {
    "[max_age: 1s]": true
  }
}

See how it incremented both the date stamp in the index name and the version number at the end? And that I didn't need to specify a target index name?

You can do likewise by creating the index & alias with the timestring in the original index creation, and then you won't have to worry about specifying the target index name. Rollover has you covered for date_math automatically.

If you do feel it necessary to manually name the target indices, I suggest adding date_math down to (at least) the hour and minute in your target index name, and omit the -1 at the end. This will ensure that the old index name and new index name will not collide (unless you try to rollover 2x in 1 minute).

Thank you that explained exactly what I was missing, it would be so very helpful if that information was in the docs.

It is in the docs. It's just not as spelled out as clearly as I just did.

Particularly, it's in the 1 and 2, blue-circle explanations under the code block:

1 Creates an index named with today’s date (e.g.) logs-2016.10.31-1
2 Rolls over to a new index with today’s date, e.g. logs-2016.10.31-000002 if run immediately, or logs-2016.11.01-000002 if run after 24 hours

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.