I run ES with default setting which takes 5 shards. And create index
continually for a night. This morning I found that the disk is full and
there are disk full exception in elasticsearch.log:
[2013-07-23 09:50:47,776][WARN ][index.merge.scheduler ] [Madam Slay]
[20130723][0] failed to merge
java.io.IOException: no space on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:499)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474)
at
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput.flushBuffer(RateLimitedFSDirectory.java:186)
at
org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:78)
at
org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:65)
at
org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at
org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
[2013-07-23 09:50:49,480][WARN ][index.shard.service ] [Madam Slay]
[20130723][0] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [20130723][0]
Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:796)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:755)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: no space on device
[2013-07-23 09:50:52,646][WARN ][cluster.action.shard ] [Madam Slay]
sending failed shard for [20130723][0], node[TjhCZBlpQu-OwOm-8CzFHA], [P],
s[STARTED], reason [engine failure, message
[MergeException[java.io.IOException: no space on device]; nested:
IOException[ no space on device]; ]]
[2013-07-23 09:50:52,646][WARN ][cluster.action.shard ] [Madam Slay]
received shard failed for [20130723][0], node[TjhCZBlpQu-OwOm-8CzFHA], [P],
s[STARTED], reason [engine failure, message
[MergeException[java.io.IOException: no space on device]; nested:
IOException[ no space on device]; ]]
[2013-07-23 10:02:02,902][WARN ][index.shard.service ] [Madam Slay]
[20130723][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [20130723][2]
Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:796)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:755)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: no space on device
Also when I try to search something in ES, it returns shard 0 is failed.
reason BroadcastShardOperationFaailedException[[20130723][0] No active
shard(s)].
Is it caused by the disk full? why other 4 shards works fine? how can I
bring this shard back to work?
there is an error message in your exception telling you 'no space on
device' - if you run out of disk space, bad things can happen... for
elasticsearch this means to fail the shard and go somewhere else with the
data (if you run a single node, this is not going to work).
The other four shards most likely still work, because you did not yet try
to index data into them, otherwise they would fail as well.
I run ES with default setting which takes 5 shards. And create index
continually for a night. This morning I found that the disk is full and
there are disk full exception in elasticsearch.log:
[2013-07-23 09:50:47,776][WARN ][index.merge.scheduler ] [Madam Slay]
[20130723][0] failed to merge
java.io.IOException: no space on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:499)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474)
at
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput.flushBuffer(RateLimitedFSDirectory.java:186)
at
org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:78)
at
org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:65)
at
org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at
org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
[2013-07-23 09:50:49,480][WARN ][index.shard.service ] [Madam Slay]
[20130723][0] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [20130723][0]
Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:796)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:755)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: no space on device
[2013-07-23 09:50:52,646][WARN ][cluster.action.shard ] [Madam Slay]
sending failed shard for [20130723][0], node[TjhCZBlpQu-OwOm-8CzFHA], [P],
s[STARTED], reason [engine failure, message
[MergeException[java.io.IOException: no space on device]; nested:
IOException[ no space on device]; ]]
[2013-07-23 09:50:52,646][WARN ][cluster.action.shard ] [Madam Slay]
received shard failed for [20130723][0], node[TjhCZBlpQu-OwOm-8CzFHA], [P],
s[STARTED], reason [engine failure, message
[MergeException[java.io.IOException: no space on device]; nested:
IOException[ no space on device]; ]]
[2013-07-23 10:02:02,902][WARN ][index.shard.service ] [Madam Slay]
[20130723][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [20130723][2]
Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:796)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:755)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: no space on device
Also when I try to search something in ES, it returns shard 0 is failed.
reason BroadcastShardOperationFaailedException[[20130723][0] No active
shard(s)].
Is it caused by the disk full? why other 4 shards works fine? how can I
bring this shard back to work?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.