Index management error: "Index is not the write index for alias"

[bartong (Grant)]

Hello all,

While trying to configure Index Lifecycle Policies for my first time, I've encountered an error that I can't resolve.

Context:

• ELK stack v7.0.1
• Single-node cluster
• All indexes were automatically created by Logstash / Filebeat / Metricbeat

I changed the Index Lifecycle Policies through the Kibana UI so that our indexes will go Hot > Cold > Delete.

In order to initiate these changes to the current indexes, I entered the following to the Elasticsearch API:

curl -X POST "localhost:9200/metricbeat-7.0.0/_rollover" -H 'Content-Type: application/json' -d'
{
  "conditions": {
   "max_age":   "7d",
   "max_size":  "10gb"
  }
}'

This command successfully rolled-over the indexes: new indexes were created and are being written to, while the old ones are not being written to.

However, the Kibana Index Policies page says that the old indexes are experiencing errors. Specifically, the errors look like this:

illegal_argument_exception: index [metricbeat-7.0.0] is not the write index for alias [metricbeat-7.0.0-2019.04.26-000001]

This page also says the following:
"Current action: rollover
Failed step: check-rollover-ready
Current phase: hot" (should be cold)

The Phase Definition, however, is the correct NEW policy that I defined through Kibana.

Kibana no longer shows any data from these older indexes, only data from the new indexes.

When I output the aliases for our indexes via the API, it says this:

"metricbeat-7.0.0-2019.04.26-000001" : {
 "aliases" : {
  "metricbeat-7.0.0" : {
   "is_write_index" : false
   }
 }
},
"metricbeat-7.0.0-2019.05.08-000002" : {
 "aliases" : {
  "metricbeat-7.0.0" : {
   "is_write_index" : true
  }
 }
},

How do I make it so that these indexes are properly rolled-over, and the data can be read alongside that of the newer indexes (eg, through "metricbeat-*")?

Edit- I forgot to say, thank you for your time.

[John_Fredrickson]

Had the same thing happen.

If you use the rollover API manually to do the rollover instead of using your ILM policy, each time the policy runs it will throw this error. This happens because the index is not being flagged as complete, when the policy runs it appears to be trying to manage the index but it's not flagged as writable.

Add "index.lifecycle.indexing_complete": "true" to the index in question, then clear the error by retrying the policy. When the policy next runs(default is every 10 minutes) it should transition from the "hot" phase to whatever phase you have defined next in your policy.

[gbrown (Gordon Brown)]

@John_Fredrickson is correct, this will happen if you use the Rollover API on an index which is managed by an ILM policy. Because the index isn't the write index for the metricbeat-7.0.0 anymore, ILM can't be sure that everything is working as intended, so it takes the "safer" option and does nothing, rather than proceed to the Warm phase when that might not be what you want. John is also correct that you can set index.lifecycle.indexing_complete to true and use the Retry API to allow ILM to skip rollover in this case.

I should note that you'll only have this issue on the one index that was rolled over manually, (metricbeat-7.0.0-2019.04.26-000001 in this case), all subsequent indices should proceed through their lifecycle policy as normal, assuming the rollover is handled by ILM.

[bartong (Grant)]

Thanks for the replies. I ended up simply deleting those initial indexes. The successive indexes did rollover properly, so all is well.