Failed to execute on node [YK3h1SM5SZW2K-JloBdVCg]

I am using curl -H "Content-Type: application/x-ndjson" -XPOST --data-binary @file command to import json file into ES cluster.
my master nodes contains : [,,]
my data nodes contains: [,,,].
the master log shows here:

failed to excute on node  [YK3h1SM5SZW2K-JloBdVCg]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [data5][][cluster:monitor/nodes/stats[n]] request_id [19314] timed out after [15001ms]
    at org.elasticsearch.transport.TransportService$
Received response for a request that has timed out, sent [25237ms] ago, timed out [10236] ago, action [cluster: monitor/nodes/stats[n]], node

THE EXCEPTIONS are random appearance, sometimes it can process success and es cluster is GREEN , sometimes some of my data nodes get crashed.

I am not sure it appeared because of low memory ,as I use free -lh to show the memory status ,found only 200MB FREE (total 64GB)

Anybody could give me some hints? :woman:t2:

That looks like monitoring failing to gather statistics about node performance in a timely fashion.
My guess is these data nodes are running hot? Are you seeing GC warnings in their log files?

I am sure I can see many gc overhead in all of my nodes log file.
And my curl bulk command process about 300MB json file at one time, should I cut it into more split file?

Yes, that would certainly help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.