Hello. We have a datastream with ILM policy (50gb or 30 days).
It works, but when rollover occurs we see some errors in a logs and some indices created!
This problem occurs only with datastreams and not with simple indices with ILM.
ElasticSearch 7.9.2
3 nodes: 2 data/master and 1 master-only
[2020-12-12T12:42:30,538][INFO ][o.e.x.i.IndexLifecycleTransition] [es5] moving index [.ds-logging_fe-000009] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [logging_fe]
[2020-12-12T12:42:31,469][INFO ][o.e.x.i.IndexLifecycleTransition] [es5] moving index [.ds-logging_fe-000010] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [logging_fe]
[2020-12-12T12:42:32,658][INFO ][o.e.x.i.IndexLifecycleTransition] [es5] moving index [.ds-logging_fe-000007] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [logging_fe]
[2020-12-12T12:42:34,424][INFO ][o.e.x.i.IndexLifecycleTransition] [es5] moving index [.ds-logging_fe-000006] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [logging_fe]
[2020-12-12T12:42:35,334][INFO ][o.e.x.i.IndexLifecycleTransition] [es5] moving index [.ds-logging_fe-000008] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [logging_fe]
[2020-12-12T12:42:36,507][INFO ][o.e.c.m.MetadataCreateIndexService] [es5] [.ds-logging_fe-000011] creating index, cause [rollover_data_stream], templates [logging_fe], shards [5]/[1]
[2020-12-12T12:42:38,176][INFO ][o.e.c.m.MetadataCreateIndexService] [es5] [.ds-logging_fe-000012] creating index, cause [rollover_data_stream], templates [logging_fe], shards [5]/[1]
[2020-12-12T12:42:38,180][ERROR][o.e.x.i.IndexLifecycleRunner] [es5] policy [logging_fe] for index [.ds-logging_fe-000010] failed on step [{"phase":"hot","action":"rollover","name":"attempt-rollover"}]. Moving to ERROR step
org.elasticsearch.ElasticsearchException: Concurrent modification of alias [logging_fe] during rollover
May be the key is that Elastic tries to create two indices at the same time, but, why?
[2020-12-12T12:42:36,507][INFO ][o.e.c.m.MetadataCreateIndexService] [es5] [.ds-logging_fe-000011] creating index, cause [rollover_data_stream], templates [logging_fe], shards [5]/[1]
[2020-12-12T12:42:38,176][INFO ][o.e.c.m.MetadataCreateIndexService] [es5] [.ds-logging_fe-000012] creating index, cause [rollover_data_stream], templates [logging_fe], shards [5]/[1]
full log file (on google drive)
Thanks!