# Metricbeat: "More than one write index" errors in version 7.13.x

**URL:** <https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355>\
**Category:** Beats\
**Tags:** ilm-index-lifecycle-management, metricbeat\
**Created:** [June 8, 2021, 10:15pm UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355 "2021-06-08T22:15:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bwright1558](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bwright1558/32/58732_2.png) [@bwright1558](https://discuss.elastic.co/u/bwright1558)\
**Post date:** [June 8, 2021, 10:15pm UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/1 "2021-06-08T22:15:23Z")

</div>

Ever since upgrading the elastic stack from 7.12.1 to 7.13.x, we've been getting frequent hang ups with running metricbeat (or any other beat) due to the following error:

```auto
2021-06-08T22:01:05.930Z	ERROR	[index-management.ilm]	ilm/std.go:128	Index Alias metricbeat-7.13.1 setup failed: failed to create alias: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.08-000001]"}],"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.08-000001]"},"status":500}: 500 Internal Server Error: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.08-000001]"}],"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.08-000001]"},"status":500}.

```

This seems to occasionally occur when we restart all our metricbeat instances.  
Here's the part of our metricbeat config I think is relevant to this issue:

```auto
setup.ilm.rollover_alias: metricbeat-%{[agent.version]}
setup.ilm.policy_name: metricbeat-policy
setup.ilm.policy_file: /usr/share/metricbeat/ilm-policy.json
setup.ilm.overwrite: true

```

And here's the corresponding ILM policy file:

```auto
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 9, 2021, 12:39am UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/2 "2021-06-09T00:39:45Z")

</div>

When that happens, can you check the output from `_cat/aliases/metricbeat*?v`?

---

<div class="post-metadata">

**Author:** ![bwright1558](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bwright1558/32/58732_2.png) [@bwright1558](https://discuss.elastic.co/u/bwright1558)\
**Post date:** [June 9, 2021, 12:59am UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/3 "2021-06-09T00:59:14Z")

</div>

Here's the output from `GET _cat/aliases/metricbeat*?v=`

```auto
alias index filter routing.index routing.search is_write_index
metricbeat-7.13.1 metricbeat-7.13.1-2021.06.03-000001 - - - true

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 9, 2021, 1:01am UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/4 "2021-06-09T01:01:16Z")

</div>

Thanks, but it'd need to be from when the error occurs, to see if there is indeed two indices attached to it.

---

<div class="post-metadata">

**Author:** ![bwright1558](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bwright1558/32/58732_2.png) [@bwright1558](https://discuss.elastic.co/u/bwright1558)\
**Post date:** [June 9, 2021, 1:23am UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/5 "2021-06-09T01:23:45Z")

</div>

The error is occurring. I'm watching the metricbeat logs and running the above aliases query as I type this message. Only one index ever shows up in the output.

---

<div class="post-metadata">

**Author:** ![bwright1558](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bwright1558/32/58732_2.png) [@bwright1558](https://discuss.elastic.co/u/bwright1558)\
**Post date:** [June 9, 2021, 3:47pm UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/6 "2021-06-09T15:47:45Z")

</div>

I found an issue on GitHub that appears to be exactly the issue I'm having:

> <https://github.com/elastic/beats/issues/26142>
>
> When ILM is enabled, libbeat will attempt to create a write alias. It does this …by creating an index, and attempting to pointing the write alias to it:
> 
> https://github.com/elastic/beats/blob/dc576280f49baec169cd86f64a043cdc3d4fbab6/libbeat/idxmgmt/ilm/client\_handler.go#L178-L208
> 
> If the \_index\_ already exists, Elasticsearch will return 400; libbeat then checks if the alias exists, and carries on without error if it does.
> 
> If the index does not exist (e.g. because it rolled over), but the \_write alias\_ exists and points to some other index, then Elasticsearch returns 500. In this case, libbeat fails index management setup with an error like (taken from apm-server):
> 
> \> Index Alias apm-7.13.0-span setup failed: failed to create alias: {"error":{"root\_cause":\[{"type":"illegal\_state\_exception","reason":"alias \[apm-7.13.0-span\] has more than one write index \[apm-7.13.0-span-000001,apm-7.13.0-span-000008\]"}\],"type":"illegal\_state\_exception","reason":"alias \[apm-7.13.0-span\] has more than one write index \[apm-7.13.0-span-000001,apm-7.13.0-span-000008\]"},"status":500}: 500 Internal Server Error: {"error":{"root\_cause":\[{"type":"illegal\_state\_exception","reason":"alias \[apm-7.13.0-span\] has more than one write index \[apm-7.13.0-span-000001,apm-7.13.0-span-000008\]"}\],"type":"illegal\_state\_exception","reason":"alias \[apm-7.13.0-span\] has more than one write index \[apm-7.13.0-span-000001,apm-7.13.0-span-000008\]"},"status":500}.

And here's the corresponding fix:

> <https://github.com/elastic/beats/pull/26146>
>
> \## What does this PR do?
> 
> When creating the initial index/write alias fails, d…on't check the status code, just check if the alias exists regardless of the error. The status code will be different (at least 400 and 500) depending on failure scenario.
> 
> \## Why is it important?
> 
> This fixes a bug where alias creation fails when the alias exists but points to another index, and the initial index does not exist, e.g. due to ILM deletion.
> 
> \## Checklist
> 
> \- \[x\] My code follows the style guidelines of this project
> \- \[x\] I have commented my code, particularly in hard-to-understand areas
> ~- \[\] I have made corresponding changes to the documentation~
> ~- \[\] I have made corresponding change to the default configuration files~
> \- \[x\] I have added tests that prove my fix is effective or that my feature works
> \- \[x\] I have added an entry in \`CHANGELOG.next.asciidoc\` or \`CHANGELOG-developer.next.asciidoc\`.
> 
> \## How to test this PR locally
> 
> 1. Run a beat with \`setup.ilm.overwrite: true\`
> 2. In Elasticsearch, force a rollover (\`POST /\<alias\>/\_rollover\`) and delete the initial index
> 3. Restart the beat; there should be no errors due to creating the write alias
> 
> \## Related issues
> 
> Closes #26142

Is there an ETA when this will be released?

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [July 5, 2021, 12:03pm UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/7 "2021-07-05T12:03:47Z")

</div>

look like this have been release with v7.13.2 version?

We see also this bug at midnight sometimes, elasticsearch logs:

```auto
alias [apm-7.13.2-span] has more than one write index [apm-7.13.2-span-000013,apm-7.13.2-span-000001]

```

APM queue become full and do not accept any new data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2021, 2:03pm UTC](https://discuss.elastic.co/t/metricbeat-more-than-one-write-index-errors-in-version-7-13-x/275355/8 "2021-08-02T14:03:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.