# Elasticsearch says the disk is out of space, but \[df -h\] shows not **URL:** \ **Category:** Elasticsearch\ **Created:** [March 9, 2018, 8:40am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223 "2018-03-09T08:40:38Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![helloEson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helloeson/32/28629_2.png) [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Post date:** [March 9, 2018, 8:40am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/1 "2018-03-09T08:40:38Z")
Today hours ago, when we found the Kibana shows an error like below: Error: Request to Elasticsearch failed: {"error":{"root\_cause":,"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":},"status":503}. So we try to find out what happened, and we found these logs: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c76d085a30402d5eeaf5c843dab1e407d91635de.png) After a day, the log shows that the space has Exhausted: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da34941327f1e2c34d651a430a92cb04ea47eeed.png) And, finally, an error occurs: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/e/be7f36ea0d5b5cc20af8d8726802171256687246.png) The words after java.io.IOException means No space left on device. But what the server says there's a lot of space left: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/4/0424b3edeb0bb123d977f6ea6822bb589f5808ea.png) Pay attension on the /data0, where we have the elasticsearch data stored on. Anybody met this scene before? Our ELK version is 5.6.4, and before this problem, it has about 7 billion docs, the total indice data size is 1.7TB. ---
**Author:** ![rmarin](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rmarin](https://discuss.elastic.co/u/rmarin)\ **Post date:** [March 9, 2018, 10:27am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/2 "2018-03-09T10:27:01Z")
Hi, how many node ES have you ? Perhaps, the disk issues is on an other node. ---
**Author:** ![helloEson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helloeson/32/28629_2.png) [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Post date:** [March 9, 2018, 10:27am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/3 "2018-03-09T10:27:48Z")
Only one node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 9, 2018, 10:28am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/4 "2018-03-09T10:28:56Z")
What type of disk is it? ---
**Author:** ![helloEson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helloeson/32/28629_2.png) [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Post date:** [March 9, 2018, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/5 "2018-03-09T10:30:44Z")
it's ext4. ---
**Author:** ![rmarin](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rmarin](https://discuss.elastic.co/u/rmarin)\ **Post date:** [March 9, 2018, 1:14pm UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/7 "2018-03-09T13:14:48Z")
Probably yes but have you restart ES ? ---
**Author:** ![helloEson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helloeson/32/28629_2.png) [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Post date:** [March 9, 2018, 3:23pm UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/8 "2018-03-09T15:23:35Z")
Yes, I just restarted ES. and it seems work again, but haven't find out the reason. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 6, 2018, 3:23pm UTC](https://discuss.elastic.co/t/elasticsearch-says-the-disk-is-out-of-space-but-df-h-shows-not/123223/9 "2018-04-06T15:23:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.