When Adding replicas I recieve `failed to list shard for shard_store on node ` warning repeatedly

we are using elasticsearch 6.3. i received this same WARN message when i restored from a snapshot and a separate time when I added extra replicas.

exception: {
   "exception_class": "org.elasticsearch.action.FailedNodeException",
   "exception_message": "Failed node [hn3cgxthRmyYMn-ZbOmuRw]"
level: WARN
logger_name: org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator
message: [base-event-ce70d90][0]: failed to list shard for shard_store on node [hn3cgxthRmyYMn- 
thread_name: elasticsearch[prod1-ip-10-185-91-176.ec2.internal][transport_client_boss][T#4]

Was this message also recorded in the Elasticsearch logs? If so, could you provide the stack trace?

Does the node UUID mentioned, hn3cgxthRmyYMn-ZbOmuRw, appear elsewhere in the logs too?

Sorry i wasnt able to find any stack trace in the elastic logs.

for that node ID hn3cgxthRmyYMn-ZbOmuRw the only other logs i see except for the "failed to list shard for shard_store on node", are the normal cluster joinging logs
added {{.....internal}{hn3cgxthRmyYMn-ZbOmuRw}{3i8y7MXaQR-kBQLVslvaaA}{.....}{....:9300}.... ml.machine_memory=32891092992, scaling=false, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [282] source [zen-disco-node-join]])

I looked up the source of this message and it's here (in v6.3.2):

The , failure); at the end means Elasticsearch definitely tried to log a stack trace. Also this message only appears in logs, so the text quoted in the OP must be the result of some post-processing of these logs. What is happening to the logs after they leave Elasticsearch? Can you get hold of the raw logs?

FailedNodeException is a very general exception that wraps around many different things, so without the stack trace showing the inner exceptions it's not really possible to describe what's going on.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.