It appears that since yesterday our site has not been able to communicate with elastic cloud. Site logs have the below exception
System.Collections.Generic.KeyNotFoundExceptionThe given key was not present in the dictionary.
System.Web.HttpUnhandledException (0x80004005): The given key was not present in the dictionary. ---> System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
and from the elastic cloud logs there is this one
[2017-03-13 20:47:31,956][DEBUG][action.admin.cluster.node.stats] failed to execute on node [xxxxx] RemoteTransportException[[instance-xxx][x.x.x.x][cluster:monitor/nodes/stats[n]]]; nested: AlreadyClosedException[this IndexReader is closed]; Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:101) at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:55) at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:438) at org.elasticsearch.search.suggest.completion.Completion090PostingsFormat.completionStats(Completion090PostingsFormat.java:330) at org.elasticsearch.index.shard.IndexShard.completionStats(IndexShard.java:765) at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:164) at org.elasticsearch.indices.IndicesService.stats(IndicesService.java:253) at org.elasticsearch.node.service.NodeService.stats(NodeService.java:158) at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:82) at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:44) at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:92) at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:230) at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:226) at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:180) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Cluster health was green. Memory pressure and Storage pressure were not high. Not knowing what caused the issue, I just restarted the cluster and the site is now able to communicate with the cloud. To prevent this happen in future, can you please guide me as to how to investigate the issue?
Thanks,
Ramya