Elasticsearch says the disk is out of space, but [df -h] shows not


(Hello Eson) #1

Today hours ago, when we found the Kibana shows an error like below:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}.

So we try to find out what happened, and we found these logs:

After a day, the log shows that the space has Exhausted:

And, finally, an error occurs:


The words after java.io.IOException means No space left on device.

But what the server says there's a lot of space left:

image

Pay attension on the /data0, where we have the elasticsearch data stored on.

Anybody met this scene before?

Our ELK version is 5.6.4, and before this problem, it has about 7 billion docs, the total indice data size is 1.7TB.


(Remi MARIN) #2

Hi,
how many node ES have you ?
Perhaps, the disk issues is on an other node.


(Hello Eson) #3

Only one node.


(Christian Dahlqvist) #4

What type of disk is it?


(Hello Eson) #5

it's ext4.


(Remi MARIN) #7

Probably yes but have you restart ES ?


(Hello Eson) #8

Yes, I just restarted ES. and it seems work again, but haven't find out the reason.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.