ILM Error

[wwalker (Walker)]

Started getting the following error for multiple indices a few days ago, not just Winlogbeat indices. I'm running Elastic Stack 7.2.0, how do I resolve this?

illegal_argument_exception: index.lifecycle.rollover_alias [winlogbeat-7.2.0] does not point to index [winlogbeat-7.2.0-2019.07.29]

[nickpeihl]

Have you created the alias correctly? The alias specified by index.lifecycle.rollover_alias must be an existing alias which points to that index before rollover.

[wwalker (Walker)]

How do I properly create the alias?

[nickpeihl]

See the relevant documentation here. https://www.elastic.co/guide/en/elasticsearch/reference/7.2/indices-aliases.html#aliases-write-index

Also, I'm curious what steps you've already taken to start ILM? Can you share what you've done so far?

[wwalker (Walker)]

I started running into RAM issues which I suspected was due to excessive indices so I just added the ILM settings to a few pipelines and created an ILM policy to cover them all in Kibana. I then found this article after you're initial post. However, when I try to create the rollover index in dev tools, as specified in the article, I get the below error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "alias [dmarcxml-rollover] has more than one write index [dmarcxml-rollver-0001,dmarcxml-rollover-2019.07.26-0001]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "alias [dmarcxml-rollover] has more than one write index [dmarcxml-rollver-0001,dmarcxml-rollover-2019.07.26-0001]"
  },
  "status": 500
}

pipeline.yml

output {
  elasticsearch {
    ilm_rollover_alias => "dmarcxml-rollover"
    ilm_pattern => "{now/d}-0001"
    ilm_policy => "Default"
}

template settings

{
    "index_patterns": ["dmarcxml-*"],
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0,
            "lifecycle.name": "Default",
            "lifecycle.rollover_alias": "dmarcxml-rollover"
        }
    },

[wwalker (Walker)]

Any ideas on how to resolve the error shown above?

[rugenl (Len Rugen)]

I think I've been where you are now

Find all indices and aliases for "dmarcxml-rollover", one alias should have "is_write_index": true. If that's not the one you want to be using you will probably have to:

1. Stop logstash to stop indexing to the "wrong" alias
2. Change the "wrong" alias "is_write_index" to false
3. Create your new index, probably where you got the error above. See this if you want to use date math
4. Check your "is_write_index", if it's right, restart logstash.

Edit: If you don't stop logstash, I think you will get "retryable errors" from the time you set the "is_write_alias" to false until you create the new index w/alias.

[wwalker (Walker)]

@rugenl

I think I've been where you are now

Find all indices and aliases for "dmarcxml-rollover", one alias should have "is_write_index": true. If that's not the one you want to be using you will probably have to:

1. Stop logstash to stop indexing to the "wrong" alias
2. Change the "wrong" alias "is_write_index" to false
3. Create your new index, probably where you got the error above. See this if you want to use date math
4. Check your "is_write_index", if it's right, restart logstash.

Edit: If you don't stop logstash, I think you will get "retryable errors" from the time you set the "is_write_alias" to false until you create the new index w/alias.

Working to try and fix winlogbeat data....

This is making my stomach sick trying to work my way around and through this because I'm not understanding it.

How do I change the wrong alias to is_write_index: false....everything I've dumped into dev tools just throws an error at me.

I've tried deleting aliases and recreating them with the is_write_index set to true and that works but then I get mapping conflicts.

[wwalker (Walker)]

Ok...I have winlogbeat pipeline configured:

      ilm_rollover_alias => "winlogbeat"
      ilm_pattern => "{now/d}-0001"
      ilm_policy => "Delete_After_30D"

When I do a GET /_alias/winlogbeat, I get below (just showing the first few values)

{
  "winlogbeat-7.2.0-2019.07.20" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
  "winlogbeat-7.2.0-2019.06.30" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
  "winlogbeat-6.3.2-2019.07.17" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
...

If I do a GET /winlogbeat, the first entry is winlogbeat-0001 and it shows is_write_index is set to true. This seems to be good to go and I am not getting any index errors on winlogbeat-0001, however, I am getting mapping conflicts.

Running GET /_template/winlogbeat, the below settings are configured. I manually created this template by exporting the template from winlogbeat then modifying the pattern and lifecycle settings before performing a PUT /template/winlogbeat

{
  "winlogbeat" : {
    "order" : 1,
    "index_patterns" : [
      "winlogbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "Delete_After_30D",
          "rollover_alias" : "winlogbeat"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",
            "agent.id",
            "agent.name",
            "agent.type",
            "agent.version",
            "client.address",
...

GET /winlogbeat-0001 shows

{
  "winlogbeat-0001" : {
    "aliases" : {
      "winlogbeat" : {
        "is_write_index" : true
      }
    },
    "mappings" : {
      "_meta" : {
        "beat" : "winlogbeat",
        "version" : "7.2.0"
      },
      "dynamic_templates" : [

Why is it using a dynamic template when it's pattern matches my template index pattern?

[rugenl (Len Rugen)]

You get mapping errors when you try to put different version beats into the same index, they change things; like host used to be a field, now it's a group of host. fields.

I let each different beat version write into a version-named inde, and will only let clients use version we bless with forced timely upgrades when we change versions.

You can then create kibana index patterns like "winlogbeat-*" to merge queries, and use similar patterns in API queries.

This should set the false value

POST _aliases
{
  "actions": {
    "add": {
      "index": "syslog-cisco-2019.07.16-000001",
      "alias": "syslog-cisco",
      "is_write_index": false
    }
  }
}

This says "add" but it would reset the value for an existing alias too.

[Stef_Nestor]

Thanks in a large part to this lovely post, we now cover these and other top ILM configuration errors on our Troubleshooting ILM blog.

Direct summarized resolutions:

index.lifecycle.rollover_alias [x] does not point to index [y]

You’ll need to check your index settings index.lifecycle.rollover_alias . Either the index is pointing to the wrong alias or the alias does not exist. You can check the latter by running Get Aliases. Here’s an example Elastic Discuss issue. You may consider Data Streams which handle this for you.

alias [x] has more than one write index [y,z]

When you run Get Aliases, you’ll notice that two indices are marked as is_write_index:true when only one should be per alias. You’ll want to toggle is_write_index:false on one of the indices via the Aliases API.