Basic Information:
I'm experiencing an issue with my on-prem Elasticsearch cluster's (8.17) Azure Blob Storage repository. The repository is configured using a container and initially shows as connected when verified via the _snapshot/_verify API. However, after one or two days, it becomes unresponsive and returns a timeout error, especially when trying to delete snapshots.
The storage remains accessible from the Elasticsearch nodes, and a telnet test confirms connectivity. Despite this, when verifying the repository again, it results in a 502 Bad Gateway error, indicating a client timeout.
We've checked with our network team, and traffic passes through a gateway. The issue seems intermittent—sometimes the repository is reachable, and other times it's not. We're unsure if this is due to a network-related problem, a keep-alive timeout, or an issue with Azure Blob Storage itself.
To diagnose this, should we analyze the issue at different network traffic levels? Also, what specific steps would you recommend to identify whether the problem is with the gateway, network configuration, or Azure Blob Storage?
Elastic Stack > Elasticsearch
Elastic Stack snapshot-and-restore elastic-stack-searchable-snapshots slm-snapshot-lifecycle-management #azure-blob-sotage