Using elasticsearch with remote storage

We have an AKS cluster in the US running an Elasticsearch pod (version 7.16.0) tied to an Azure File Share storage account in Europe. We are getting continuous issues in the Elasticsearch logs similar to the below upon a clean startup, and the pod is basically unresponsive (e.g. create index also times out):

{"type": "server", "timestamp": "2022-04-29T15:50:22,390Z", "level": "WARN", "component": "o.e.g.PersistedClusterStateService", "": "search-123", "": "search-123-es-master-0", "message": "writing cluster state took [15422ms] which is above the warn threshold of [10s]; wrote full state with [1] indices" }

{"type": "server", "timestamp": "2022-04-29T15:51:00,933Z", "level": "ERROR", "component": "o.e.x.m.e.l.LocalExporter", "": "search-123", "": "search-123-es-master-0", "message": "failed to set monitoring template [.monitoring-alerts-7]", "cluster.uuid": "lIkSuRkeRoucLcK7W3tvBw", "": "vBeM0YOUSLOQR4wi7eyRIg" , 
"stacktrace": ["**org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (create-index-template [.monitoring-alerts-7], cause [api]) within 30s**",
"at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0( [elasticsearch-7.16.0.jar:7.16.0]",
"at java.util.ArrayList.forEach( [?:?]",
"at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1( [elasticsearch-7.16.0.jar:7.16.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ [elasticsearch-7.16.0.jar:7.16.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$ [?:?]",
"at [?:?]"] }

Is this due to the fact we're using remote storage for Elasticsearch? Is this not valid?

Other notes are that we are using a single node cluster, we've tried the workaround for Azure File Share (SMB workaround), and we've tried upgrading to Azure's premium storage for better latency.

Welcome to our community! :smiley:

Yes, and yes. This is not supported and likely to cause a bunch of issues.

1 Like

Thanks @warkolm for the welcome and quick response! We will look into alternative configurations.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.