Using ILM: error's on more then one write index

Elastic Stack Logstash
1

Greetings,

Currently i am in the process of migrating our ELK instance from manual index deletion to using ILM policy, however filebeat is sending logs to logstash but the following error is shown in the logs of logstash.

[2021-05-12T13:30:21,465][INFO ][logstash.outputs.elasticsearch][main][302547529675efa56b2617eeccc5f5889126d8038c3689d20a6facb0b903bd9a] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"alias [filebeat_alias] has more than one write index [filebeat-2021.05.12-000001,filebeat-2021.05.10]"})

i have done the following steps:

  1. Stop ILM
    POST /_ilm/stop

  2. Removed all old index from index management -> indices

  3. Add new index using date math
    PUT %filebeat-%7Bnow%2Fd%7D-000001%3E

  4. Update the alias to correct the write_alias pointer
    POST /_aliases { "actions" : [ { "add" : { "index" : "filebeat-2021.05.12-000001", "alias" : "filebeat_alias", "is_write_index" : true } } ] }

  5. Start ILM
    POST /_ilm/start

I created a policy before step 1 with the name "filebeat_policy"

PUT _ilm/policy/filebeat_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

and a index template as follows:

PUT _index_template/filebeat
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "filebeat_policy",
          "rollover_alias": "filebeat_alias"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
    "aliases": {
      "filebeat_alias": {
        "is_write_index": true
      }
    }
  },
  "index_patterns": [
    "filebeat-*"
  ],
  "composed_of": []
}

i tried it without the filebeat-2021.05.12-000001 but it indexes only one date and gives the next error: illegal_argument_exception: index name [filebeat-2021.05.11] does not match pattern ‘^.*-\d+$’

Could someone help me a step further? or point me in the right direction.

Kind regards

2

Welcome to our community! :smiley:

What does your Logstash output look like?
What is the output from _cat/indices/file*?v and _cat/aliases?v?

3

Thanks!

_cat/indices/file*?v results in the following:

health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   filebeat-2021.05.12-000001 zLNHmhfZQ72HZztjyUpzTg   1   0          0            0       208b           208b

_cat/aliases?v

results in the following: 
.kibana-event-log-7.11.2    .kibana-event-log-7.11.2-000001 -      -             -              false
.kibana-event-log-7.9.3     .kibana-event-log-7.9.3-000005  -      -             -              false
.kibana-event-log-7.8.1     .kibana-event-log-7.8.1-000010  -      -             -              true
.kibana-event-log-7.8.1     .kibana-event-log-7.8.1-000007  -      -             -              false
.kibana-event-log-7.11.0    .kibana-event-log-7.11.0-000003 -      -             -              false
filebeat_alias              filebeat-2021.05.12-000001      -      -             -              true
.kibana-event-log-7.8.0     .kibana-event-log-7.8.0-000011  -      -             -              true
.kibana-event-log-7.10.2    .kibana-event-log-7.10.2-000004 -      -             -              true
.kibana-event-log-7.10.1    .kibana-event-log-7.10.1-000006 -      -             -              true
.kibana-event-log-7.12.0    .kibana-event-log-7.12.0-000001 -      -             -              false
.kibana-event-log-7.10.2    .kibana-event-log-7.10.2-000003 -      -             -              false
.kibana-event-log-7.8.0     .kibana-event-log-7.8.0-000009  -      -             -              false
.kibana_task_manager        .kibana_task_manager_7.12.1_001 -      -             -              -
.kibana_task_manager_7.12.1 .kibana_task_manager_7.12.1_001 -      -             -              -
.kibana-event-log-7.8.0     .kibana-event-log-7.8.0-000010  -      -             -              false
ilm-history-2               ilm-history-2-000009            -      -             -              false
.kibana-event-log-7.9.3     .kibana-event-log-7.9.3-000004  -      -             -              false
.kibana-event-log-7.11.1    .kibana-event-log-7.11.1-000002 -      -             -              false
.kibana-event-log-7.10.1    .kibana-event-log-7.10.1-000005 -      -             -              false
ilm-history-2               ilm-history-2-000008            -      -             -              false
.kibana-event-log-7.10.1    .kibana-event-log-7.10.1-000003 -      -             -              false
.kibana-event-log-7.11.1    .kibana-event-log-7.11.1-000001 -      -             -              false
.kibana-event-log-7.8.1     .kibana-event-log-7.8.1-000008  -      -             -              false
.kibana-event-log-7.8.1     .kibana-event-log-7.8.1-000009  -      -             -              false
ilm-history-3               ilm-history-3-000004            -      -             -              false
.kibana-event-log-7.9.2     .kibana-event-log-7.9.2-000008  -      -             -              true
.kibana-event-log-7.12.1    .kibana-event-log-7.12.1-000001 -      -             -              true
.kibana_7.12.0              .kibana_7.12.0_001              -      -             -              -
.kibana-event-log-7.9.2     .kibana-event-log-7.9.2-000006  -      -             -              false
.kibana-event-log-7.11.0    .kibana-event-log-7.11.0-000001 -      -             -              false
.kibana-event-log-7.10.2    .kibana-event-log-7.10.2-000001 -      -             -              false
.kibana-event-log-7.11.0    .kibana-event-log-7.11.0-000002 -      -             -              false
.kibana-event-log-7.10.0    .kibana-event-log-7.10.0-000007 -      -             -              true
.kibana-event-log-7.12.0    .kibana-event-log-7.12.0-000002 -      -             -              true
.kibana-event-log-7.11.0    .kibana-event-log-7.11.0-000004 -      -             -              true
.kibana-event-log-7.11.2    .kibana-event-log-7.11.2-000003 -      -             -              true
.kibana-event-log-7.9.2     .kibana-event-log-7.9.2-000005  -      -             -              false
.kibana-event-log-7.8.0     .kibana-event-log-7.8.0-000008  -      -             -              false
.kibana                     .kibana_7.12.1_001              -      -             -              -
.kibana_7.12.1              .kibana_7.12.1_001              -      -             -              -
.kibana_task_manager_7.12.0 .kibana_task_manager_7.12.0_001 -      -             -              -
.kibana-event-log-7.11.2    .kibana-event-log-7.11.2-000002 -      -             -              false
.kibana-event-log-7.10.1    .kibana-event-log-7.10.1-000004 -      -             -              false
.kibana-event-log-7.9.3     .kibana-event-log-7.9.3-000006  -      -             -              false
.kibana-event-log-7.10.0    .kibana-event-log-7.10.0-000006 -      -             -              false
.kibana-event-log-7.9.2     .kibana-event-log-7.9.2-000007  -      -             -              false
.kibana-event-log-7.9.3     .kibana-event-log-7.9.3-000007  -      -             -              true
.kibana-event-log-7.10.0    .kibana-event-log-7.10.0-000004 -      -             -              false
.kibana-event-log-7.10.2    .kibana-event-log-7.10.2-000002 -      -             -              false
.kibana-event-log-7.10.0    .kibana-event-log-7.10.0-000005 -      -             -              false
.kibana-event-log-7.11.1    .kibana-event-log-7.11.1-000003 -      -             -              true
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

5

For future note when searched, this is now answered in the blog Troubleshooting ILM blog & direct summarized discussed error resolutions are:

alias [x] has more than one write index [y,z]

When you run Get Aliases, you’ll notice that two indices are marked as is_write_index:true when only one should be per alias. You’ll want to toggle is_write_index:false on one of the indices via the Aliases API.

index name [x] does not match pattern *^.*-\d+

The index name’s regex pattern matching is a prerequisite for rollover to work. The most common implication users miss is not realizing the index name needs to end with trailing digits, e.g. my-index-000001 and instead only use my-index which does match the pattern requirement. Here’s an example Elastic Discuss issue. You may consider Data Streams which handle this for you.