I am getting sporadic failures from Elasticsearch. I've seen these two errors:
Primary shard is not active or isn't assigned to a known node. Timeout: [1m]
and
NoShardAvailableActionException
But the cluster state is green, and all shards are assigned. There's no errors in the logfile. I was on 5.3.2 and upgraded to 5.4.0, same issue. There is enough RAM free, and over 70% disk free. Both nodes can connect to each other on port 9300.
How should I go about troubleshooting this? It doesn't happen on-demand - sometimes the same index request will fail then succeed a minute later.
Health:
{
"cluster_name" : "nx-stg1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 9,
"active_shards" : 12,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}