When Adding replicas I recieve `failed to list shard for shard_store on node ` warning repeatedly

Elastic Stack Elasticsearch
1

we are using elasticsearch 6.3. i received this same WARN message when i restored from a snapshot and a separate time when I added extra replicas.

exception: {
   "exception_class": "org.elasticsearch.action.FailedNodeException",
   "exception_message": "Failed node [hn3cgxthRmyYMn-ZbOmuRw]"
}
level: WARN
logger_name: org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator
message: [base-event-ce70d90][0]: failed to list shard for shard_store on node [hn3cgxthRmyYMn- 
    ZbOmuRw]
thread_name: elasticsearch[prod1-ip-10-185-91-176.ec2.internal][transport_client_boss][T#4]
2

Was this message also recorded in the Elasticsearch logs? If so, could you provide the stack trace?

Does the node UUID mentioned, hn3cgxthRmyYMn-ZbOmuRw, appear elsewhere in the logs too?

3

Sorry i wasnt able to find any stack trace in the elastic logs.

for that node ID hn3cgxthRmyYMn-ZbOmuRw the only other logs i see except for the "failed to list shard for shard_store on node", are the normal cluster joinging logs
i.e.
added {{.....internal}{hn3cgxthRmyYMn-ZbOmuRw}{3i8y7MXaQR-kBQLVslvaaA}{.....}{....:9300}.... ml.machine_memory=32891092992, scaling=false, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [282] source [zen-disco-node-join]])

4

I looked up the source of this message and it's here (in v6.3.2):

The , failure); at the end means Elasticsearch definitely tried to log a stack trace. Also this message only appears in logs, so the text quoted in the OP must be the result of some post-processing of these logs. What is happening to the logs after they leave Elasticsearch? Can you get hold of the raw logs?

FailedNodeException is a very general exception that wraps around many different things, so without the stack trace showing the inner exceptions it's not really possible to describe what's going on.