Metricbeat not creating Index correctly

Elastic Stack Beats
1

I am running Stack version 7.13.4 with Metricbeat

Everything seemed to be working fine, then we wanted to remove all the metricbeat data and start over. So we stopped metricbeat, and deleted all of the Metricbeat indexes. When we restarted metricbeat the index was created but not as it was before.

Prior we had index names like: metricbeat-7.13.4-2021-10-04-000001
But after the restart, they are named as: metricbeat-7.13.4 (no date, no counter)

Looking in the metricbeat logs I see:

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch([http://192.168.123.121:9200](http://192.168.123.121:9200/))): Connection marked as failed because the onConnect callback failed: resource 'metricbeat-7.13.4' exists, but it is not an alias

Any ideas why this is?

THanks,

Michael

2

Hmmm could you try ./metricbeat setup command to see if that will help? Thanks.

1 Like
3

So when you did that you deleted the write alias metricbeat-7.13.4 as well besides the actual indexes with the data in it. You don't want to do that.

When you initially ran setup the metricbeat-7.13.4 write alias was created it looked something like this you can read about them here. They are part of ILM. running setup will re-create it.

  "metricbeat-7.13.4-2021.10.07-000001" : {
    "aliases" : {
      "metricbeat-7.13.4" : {
        "is_write_index" : true
      }

When working properly, metricbeat writes to metricbeat-7.13.4 which is alias to the real index metricbeat-7.13.4-2021.10.07-000001

When the index rolls over another write alias is created and so on.

  "metricbeat-7.13.4-2021.10.08-000002" : {
    "aliases" : {
      "metricbeat-7.13.4" : {
        "is_write_index" : true
      }

(and the old one is no longer the write alias. "is_write_index" : false

When you deleted the alias and restarted metricbeat now the docs get written to metricbeat-7.13.4 because it is no longer aliased and it will no longer roll over etc.

Unfortunately you will need to delete that index and then run set up again.

You need to delete that new real index because its named the same as the write index... A case of name collision.

Need to clean it up, And then be careful not to delete it in the future.
:slight_smile:

1 Like
4

Thanks!, You are exactly correct. I was confused when I stopped metricbeat, deleted the index, restarted metricbeat and the index didn't come back as expected. The problem was there was another instance of metricbeat running somewhere else and as soon as I deleted the index, it was recreated (incorrectly again) by the other instance of metricbeat. Once I stopped both instances, removed the index and the restart metricbeat, the index came back correctly.

Thanks for the Help!

1 Like
5

Ohh yes I have run into that myself!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.