For some reason, Stack Monitoring is now showing "Total shards: 0", "Documents: 0", "Data: 0.0B", and the node count is not including my data nodes. Why is no data present for any of my data nodes? If I restart Elasticsearch on one of the data nodes, it will briefly increase the node count by 1, but it then returns back to not including that node. I have seen a few suggestions to check the time on the servers, and they are all correct. Not sure where else to look to figure this out.
Elasticsearch 7.6.0
Kibana 7.6.0
GET _cluster/health?pretty
{
"cluster_name" : "Cluster-Name",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 9,
"number_of_data_nodes" : 5,
"active_primary_shards" : 360,
"active_shards" : 720,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
GET _nodes/stats
{
"_nodes" : {
"total" : 9,
"successful" : 4,
"failed" : 5,
"failures" : [
{
"type" : "failed_node_exception",
"reason" : "Failed node [NzA6DLQoSvyYPZpC1_nc_w]",
"node_id" : "NzA6DLQoSvyYPZpC1_nc_w",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-410]"
}
},
{
"type" : "failed_node_exception",
"reason" : "Failed node [vzLH-Rc3RtqbFVnh6j39Cw]",
"node_id" : "vzLH-Rc3RtqbFVnh6j39Cw",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-2449]"
}
},
{
"type" : "failed_node_exception",
"reason" : "Failed node [mVi86ywZQ8e8cWJ2ype9cQ]",
"node_id" : "mVi86ywZQ8e8cWJ2ype9cQ",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-116]"
}
},
{
"type" : "failed_node_exception",
"reason" : "Failed node [B9I4onxkSRCHnQsl_VdX2Q]",
"node_id" : "B9I4onxkSRCHnQsl_VdX2Q",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-62]"
}
},
{
"type" : "failed_node_exception",
"reason" : "Failed node [rnj8pp48S1mX2fNej_TM-w]",
"node_id" : "rnj8pp48S1mX2fNej_TM-w",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-227]"
}
}
]
},
"cluster_name" : "Cluster-Name",
"nodes" : {
THE FOUR UP NODES
}
}
When I look in Kibana, it shows 4 nodes (3x master and 1x coordinator), and no counts for shards, nodes, data, etc.. All the nodes do appear listed properly as up on the bottom of the page.