MetricBeat no longer creating new indexes

Hello. I've been using MetricBeat for a while now and it was working just fine. All of a sudden, about mid April, I noticed that it stopped logging to the index. Upon closer inspection, all of the metricbeat agents are still running. However, I'm getting this error on the Elastic server:

internal metricbeat[27650]: 2022-05-16T14:14:07.947-0400 ERROR [Elasticsearch] Elasticsearch/client.go:213 Failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no write index is defined for alias [metricbeat-7.7.0]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"}],"type":"illegal_argument_exception","reason":"no write index is defined for alias [metricbeat-7.7.0]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"},"status":400}

Sure enough, once I looked, the last index was created on 3/19 (currently "warm" phase). When I installed this, I believe it automatically created a ILP. I looked and it is set to only keep indexes in the "hot" phase (write index) for 30 days. Then, it usually rolls them over to warm and creates a new index. I have the delete phase set to 90 days old.

I'm not sure what happened, but does anyone know how I can manually create a new MetricBeat index and set it as the "write" index? I'm assuming once this is out there, it will automatically start using it?

Thanks!

Have you tried fixing your issue by following the guidance provided here: There are some way to fix is_write_index=false for all my indexes? - #2 by gbrown

I figured out how to fix this in the meantime. I basically did the following.

Created a new index using the API (following naming convention):

PUT /metricbeat-7.7.0-2022.05.21-000024
{
  "aliases": {
    "metricbeat-7.7.0": {
      "is_write_index": true
    }
  },
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "blocks": {
        "write": "true"
      },
"priority": "50",
      "number_of_replicas": "1",
      "lifecycle": {
        "name": "metricbeat",
        "rollover_alias": "metricbeat-7.7.0",
        "indexing_complete": "false"
      },
      "codec": "best_compression",
      "number_of_shards": "1"
    }
  }
}

This got the index into a "rollover" status with a "hot" phase. However, index was still not getting any new documents. Upon looking at the logs I saw this:

"elastic FORBIDDEN/8/index write"

After doing some research, I was not having any issues with disk space or memory. So, I updated the index with this:

PUT /metricbeat-7.7.0-2022.05.21-000024/_settings
{
  "index.blocks.read_only_allow_delete": null,
  "index.blocks.write": null
}

That got the data flowing into this new index! So, I think I'm ok now. Just not sure why this stopped indexing in the first place.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.