ILM Error

Started getting the following error for multiple indices a few days ago, not just Winlogbeat indices. I'm running Elastic Stack 7.2.0, how do I resolve this?

illegal_argument_exception: index.lifecycle.rollover_alias [winlogbeat-7.2.0] does not point to index [winlogbeat-7.2.0-2019.07.29]

image

Have you created the alias correctly? The alias specified by index.lifecycle.rollover_alias must be an existing alias which points to that index before rollover.

How do I properly create the alias?

See the relevant documentation here. https://www.elastic.co/guide/en/elasticsearch/reference/7.2/indices-aliases.html#aliases-write-index

Also, I'm curious what steps you've already taken to start ILM? Can you share what you've done so far?

I started running into RAM issues which I suspected was due to excessive indices so I just added the ILM settings to a few pipelines and created an ILM policy to cover them all in Kibana. I then found this article after you're initial post. However, when I try to create the rollover index in dev tools, as specified in the article, I get the below error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "alias [dmarcxml-rollover] has more than one write index [dmarcxml-rollver-0001,dmarcxml-rollover-2019.07.26-0001]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "alias [dmarcxml-rollover] has more than one write index [dmarcxml-rollver-0001,dmarcxml-rollover-2019.07.26-0001]"
  },
  "status": 500
}

pipeline.yml

output {
  elasticsearch {
    ilm_rollover_alias => "dmarcxml-rollover"
    ilm_pattern => "{now/d}-0001"
    ilm_policy => "Default"
}

template settings

{
    "index_patterns": ["dmarcxml-*"],
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0,
            "lifecycle.name": "Default",
            "lifecycle.rollover_alias": "dmarcxml-rollover"
        }
    },

Any ideas on how to resolve the error shown above?

I think I've been where you are now :slight_smile:

Find all indices and aliases for "dmarcxml-rollover", one alias should have "is_write_index": true. If that's not the one you want to be using you will probably have to:

  1. Stop logstash to stop indexing to the "wrong" alias
  2. Change the "wrong" alias "is_write_index" to false
  3. Create your new index, probably where you got the error above. See this if you want to use date math
  4. Check your "is_write_index", if it's right, restart logstash.

Edit: If you don't stop logstash, I think you will get "retryable errors" from the time you set the "is_write_alias" to false until you create the new index w/alias.

Working to try and fix winlogbeat data....

This is making my stomach sick trying to work my way around and through this because I'm not understanding it.

How do I change the wrong alias to is_write_index: false....everything I've dumped into dev tools just throws an error at me.

I've tried deleting aliases and recreating them with the is_write_index set to true and that works but then I get mapping conflicts.

Ok...I have winlogbeat pipeline configured:

      ilm_rollover_alias => "winlogbeat"
      ilm_pattern => "{now/d}-0001"
      ilm_policy => "Delete_After_30D"

When I do a GET /_alias/winlogbeat, I get below (just showing the first few values)

{
  "winlogbeat-7.2.0-2019.07.20" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
  "winlogbeat-7.2.0-2019.06.30" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
  "winlogbeat-6.3.2-2019.07.17" : {
    "aliases" : {
      "winlogbeat" : { }
    }
  },
...

If I do a GET /winlogbeat, the first entry is winlogbeat-0001 and it shows is_write_index is set to true. This seems to be good to go and I am not getting any index errors on winlogbeat-0001, however, I am getting mapping conflicts.

Running GET /_template/winlogbeat, the below settings are configured. I manually created this template by exporting the template from winlogbeat then modifying the pattern and lifecycle settings before performing a PUT /template/winlogbeat

{
  "winlogbeat" : {
    "order" : 1,
    "index_patterns" : [
      "winlogbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "Delete_After_30D",
          "rollover_alias" : "winlogbeat"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",
            "agent.id",
            "agent.name",
            "agent.type",
            "agent.version",
            "client.address",
...

GET /winlogbeat-0001 shows

{
  "winlogbeat-0001" : {
    "aliases" : {
      "winlogbeat" : {
        "is_write_index" : true
      }
    },
    "mappings" : {
      "_meta" : {
        "beat" : "winlogbeat",
        "version" : "7.2.0"
      },
      "dynamic_templates" : [

Why is it using a dynamic template when it's pattern matches my template index pattern?

You get mapping errors when you try to put different version beats into the same index, they change things; like host used to be a field, now it's a group of host. fields.

I let each different beat version write into a version-named inde, and will only let clients use version we bless with forced timely upgrades when we change versions.

You can then create kibana index patterns like "winlogbeat-*" to merge queries, and use similar patterns in API queries.

This should set the false value

POST _aliases
{
  "actions": {
    "add": {
      "index": "syslog-cisco-2019.07.16-000001",
      "alias": "syslog-cisco",
      "is_write_index": false
    }
  }
}

This says "add" but it would reset the value for an existing alias too.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.