Search_phase_execution_exception: [no_shard_available_action_exception] Reason: null; [no_shard_available_action_exception] Reason: null (500)

Hi
after i've receive disk full i try to remove some indices from this path:
/opt/elasticsearch/var/lib/elasticsearch/indices/

after that APM dashboard not load and give below errors. seems some indices that related to APM removed.
is there any way to fix it?

Field span.name was not found
Edit in Lens editor to see more errors

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 7,
    "successful": 5,
    "skipped": 4,
    "failed": 2,
    "failures": [
      {
        "shard": 0,
        "index": ".ds-traces-apm-default-2023.06.26-000006",
        "node": null,
        "reason": {
          "type": "no_shard_available_action_exception",
          "reason": null,
          "index_uuid": "7J_sRZeiT-W_m_171pp5vA",
          "shard": "0",
          "index": ".ds-traces-apm-default-2023.06.26-000006"
        }
      },
      {
        "shard": 0,
        "index": ".ds-traces-apm-default-2023.07.01-000007",
        "node": null,
        "reason": {
          "type": "no_shard_available_action_exception",
          "reason": null,
          "index_uuid": "8-O5Rfi4SSeH4wTSy_HgZw",
          "shard": "0",
          "index": ".ds-traces-apm-default-2023.07.01-000007"
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}
Error
search_phase_execution_exception: [no_shard_available_action_exception] Reason: null; [no_shard_available_action_exception] Reason: null (500)
URL
http://192.168.1.1:5601/internal/apm/services/use/throughput?environment=ENVIRONMENT_ALL&kuery=service.target.type%20%3A%20%22&start=2023-06-09T20%3A30%3A00.000Z&end=2023-07-10T08%3A46%3A18.746Z&transactionType=messaging&offset=2636178740ms&_inspect=true

any idea?
Thanks

Unfortunately this may have unrecoverably broken your cluster. See these docs for more information:

WARNING: Don’t modify anything within the data directory or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the data directory, then Elasticsearch may fail, reporting corruption or other data inconsistencies, or may appear to work correctly having silently lost some of your data.

After several workaround remove data stream indice (trace-apm-default), then restart apm-server.
Elastic create it with good health condition.
Issue resolved.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.