Kibana data node monitoring stops after five minutes

I've a cluster with five nodes, running 8.4.1.
One of the nodes had write rejections. We restarted the node and everything went back to a working state.
But after five minutes no monitoring data was displayed anymore.
If I restart the node monitoring data is coming in but after five minutes it stops again (short time after the dashboard shows 0 unassigned shards).

Just to mention: we had to delete some indices today because we were nearly running out of disk space (e.g. .ds-filebeat...).

It looks like the cluster is still working as expected.
If I do a _cat/nodes?v I could see the node and the data (like CPU) changes.
Kibana reports 11 nodes but only shows 10 in the list.

If I click on enter setup mode, the node is shown but greyed out.
All nodes are marked as self monitored.

Does someone know what is broken and how to fix it?

Thanks for reaching out, @mbby. I have a follow up question, did you get any error messages? If, so what did they say?

Yes. Just found the log. It says:
failed to flush export bulks

Just seen: this node only has replicas. Don't know why.

The system recovered itself. I reduced the load. But it took some hours.

1 Like

Thanks for sharing your solution, @mbby.