MetricBeat no longer creating new indexes

[ozonshak]

Hello. I've been using MetricBeat for a while now and it was working just fine. All of a sudden, about mid April, I noticed that it stopped logging to the index. Upon closer inspection, all of the metricbeat agents are still running. However, I'm getting this error on the Elastic server:

internal metricbeat[27650]: 2022-05-16T14:14:07.947-0400 ERROR [Elasticsearch] Elasticsearch/client.go:213 Failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no write index is defined for alias [metricbeat-7.7.0]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"}],"type":"illegal_argument_exception","reason":"no write index is defined for alias [metricbeat-7.7.0]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"},"status":400}

Sure enough, once I looked, the last index was created on 3/19 (currently "warm" phase). When I installed this, I believe it automatically created a ILP. I looked and it is set to only keep indexes in the "hot" phase (write index) for 30 days. Then, it usually rolls them over to warm and creates a new index. I have the delete phase set to 90 days old.

I'm not sure what happened, but does anyone know how I can manually create a new MetricBeat index and set it as the "write" index? I'm assuming once this is out there, it will automatically start using it?

Thanks!

[kvch (Noémi Ványi)]

Have you tried fixing your issue by following the guidance provided here: There are some way to fix is_write_index=false for all my indexes? - #2 by gbrown

[ozonshak]

I figured out how to fix this in the meantime. I basically did the following.

Created a new index using the API (following naming convention):

PUT /metricbeat-7.7.0-2022.05.21-000024
{
  "aliases": {
    "metricbeat-7.7.0": {
      "is_write_index": true
    }
  },
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "blocks": {
        "write": "true"
      },
"priority": "50",
      "number_of_replicas": "1",
      "lifecycle": {
        "name": "metricbeat",
        "rollover_alias": "metricbeat-7.7.0",
        "indexing_complete": "false"
      },
      "codec": "best_compression",
      "number_of_shards": "1"
    }
  }
}

This got the index into a "rollover" status with a "hot" phase. However, index was still not getting any new documents. Upon looking at the logs I saw this:

"elastic FORBIDDEN/8/index write"

After doing some research, I was not having any issues with disk space or memory. So, I updated the index with this:

PUT /metricbeat-7.7.0-2022.05.21-000024/_settings
{
  "index.blocks.read_only_allow_delete": null,
  "index.blocks.write": null
}

That got the data flowing into this new index! So, I think I'm ok now. Just not sure why this stopped indexing in the first place.