Elasticsearch says the disk is out of space, but [df -h] shows not

[helloEson]

Today hours ago, when we found the Kibana shows an error like below:

Error: Request to Elasticsearch failed: {"error":{"root_cause":,"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":},"status":503}.

So we try to find out what happened, and we found these logs:

After a day, the log shows that the space has Exhausted:

And, finally, an error occurs:

The words after java.io.IOException means No space left on device.

But what the server says there's a lot of space left:

Pay attension on the /data0, where we have the elasticsearch data stored on.

Anybody met this scene before?

Our ELK version is 5.6.4, and before this problem, it has about 7 billion docs, the total indice data size is 1.7TB.

[rmarin (Remi MARIN)]

Hi,
how many node ES have you ?
Perhaps, the disk issues is on an other node.

[helloEson]

Only one node.

[Christian_Dahlqvist]

What type of disk is it?

[helloEson]

it's ext4.

[rmarin (Remi MARIN)]

Probably yes but have you restart ES ?

[helloEson]

Yes, I just restarted ES. and it seems work again, but haven't find out the reason.