Ilm - index creation time picked as future date and time

Elastic Stack Elasticsearch
1

ILM rollover stopped working from 12/27/2020, because of wrong index creation time.

we verified system date, system date was correct during index creation.

we could confirm by seeing the elasticsearch log that system date is 2020-12-27 when index is created but index name created with future date prod-usw-es2-1-kpr2] [apache-prod-usw-es2-1-2021.12.27-000005

we use parse_origination_date index settings

[2020-12-27T11:18:32,455][INFO ][o.e.c.m.MetadataCreateIndexService] [prod-usw-es2-1-kpr2] [apache-prod-usw-es2-1-2021.12.27-000005] creating index, cause [rollover_index], templates [apache_template], shards [10]/[1]

https://192.168.26.188:9200/_cat/indices?v | grep apache
green open apache-prod-usw-es2-1-2020.12.23-000001 10 1 1945291 0 5.3gb 2.6gb
green open apache-prod-usw-es2-1-2021.12.27-000005 10 1 67599063 0 173.2gb 86.5gb
green open apache-prod-usw-es2-1-2020.12.26-000004 10 1 8103482 0 20.9gb 10.4gb
green open apache-prod-usw-es2-1-2020.12.24-000002 10 1 5238451 0 13.6gb 6.8gb
green open apache-prod-usw-es2-1-2020.12.25-000003 10 1 5115778 0 13.3gb 6.6gb

rollover didn't happened based on index size

ilm policy    
{
"ilm_apache" : {
"version" : 2,
"modified_date" : "2020-12-23T22:04:34.558Z",
"policy" : {
  "phases" : {
    "hot" : {
      "min_age" : "0ms",
      "actions" : {
        "rollover" : {
          "max_size" : "400gb",
          "max_age" : "1d"
        },
        "set_priority" : {
          "priority" : 100
        }
      }
    },
    "delete" : {
      "min_age" : "30d",
      "actions" : { }
    },
    "warm" : {
      "min_age" : "15d",
      "actions" : {
        "allocate" : {
          "include" : { },
          "exclude" : { },
          "require" : {
            "data" : "warm"
          }
        },
        "set_priority" : {
          "priority" : 50
        }
      }
    }
  }
}
}
}

index settings and we use parse_origination_date index settings

index settings  
{
"apache-prod-usw-es2-1-2021.12.27-000005" : {
"settings" : {
  "index" : {
    "lifecycle" : {
      "name" : "ilm_apache",
      "parse_origination_date" : "true",
      "rollover_alias" : "ilm_apache",
      "origination_date" : "1640563200000"
    },
    "routing" : {
      "allocation" : {
        "require" : {
          "data" : "hot"
        }
      }
    },
    "mapping" : {
      "total_fields" : {
        "limit" : "5000"
      }
    },
    "refresh_interval" : "5s",
    "number_of_shards" : "10",
    "provided_name" : "<apache-prod-usw-es2-1-{now/d{YYYY.MM.dd}}-000005>",
    "creation_date" : "1609067912453",
    "unassigned" : {
      "node_left" : {
        "delayed_timeout" : "5m"
      }
    },
    "number_of_replicas" : "1",
    "uuid" : "u4ez3WnASRWe7IT9qN1t0A",
    "version" : {
      "created" : "7090099"
    }
  }
 }
}
}
2

Thanks for reporting this Kannan (Raj?),

I think this might be a bug, would you mind opening an issue at https://github.com/elastic/elasticsearch/issues and attaching as much info as you can share? (logs, ILM explain, the index settings you posted above)

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.