Elasticsearch says the disk is out of space, but [df -h] shows not

Elastic Stack Elasticsearch
1

Today hours ago, when we found the Kibana shows an error like below:

Error: Request to Elasticsearch failed: {"error":{"root_cause":,"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":},"status":503}.

So we try to find out what happened, and we found these logs:

image
image.png1253×36 3.61 KB

After a day, the log shows that the space has Exhausted:

image
image.png1258×38 3.81 KB

And, finally, an error occurs:

image
The words after java.io.IOException means No space left on device.

But what the server says there's a lot of space left:

image

Pay attension on the /data0, where we have the elasticsearch data stored on.

Anybody met this scene before?

Our ELK version is 5.6.4, and before this problem, it has about 7 billion docs, the total indice data size is 1.7TB.

2

Hi,
how many node ES have you ?
Perhaps, the disk issues is on an other node.

3

Only one node.

4

What type of disk is it?

5

it's ext4.

7

Probably yes but have you restart ES ?

8

Yes, I just restarted ES. and it seems work again, but haven't find out the reason.