Failed to perform indices/index/shard/index on replica Index Shard

Elastic Stack Elasticsearch
1

Hi all,

I looked around and couldn't find an answer for this weird exception I've been getting half-way through the indexing of about 1M documents(each one has an ID field and a phonenumbers field):

Failed to perform indices/index/shard/index on replica Index Shard [panda][2]org.elasticsearch.transport.RemoteTransportException: [Quicksilver][inet[/192.168.7.216:9300]][indices/index/shard/index/repl
ica]Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: [panda][2] Index failed for [phonenumber#9-25987-27d8a1
ce-98c7-456b-a]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:329)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:284)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnReplica(TransportIndexAction.java:227) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHan
dler.messageReceived(TransportShardReplicationOperationAction.java:202) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHan
dler.messageReceived(TransportShardReplicationOperationAction.java:191)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.flushBuffer(SimpleFSDirectory.java:187)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:77)
at org.elasticsearch.index.store.support.AbstractStore$StoreIndexOutput.writeBytes(AbstractStore.java:350)

   at org.apache.lucene.store.IndexOutput.writeBytes(IndexOutput.java:43)
    at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:66)
    at org.apache.lucene.index.FieldsWriter.flushDocument(FieldsWriter.java:129)
    at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:149)
    at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
    at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
    at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
    at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
    at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:802)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1998)
    at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:391)
    at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:322)
    ... 8 more

[2011-04-10 20:27:33,959][WARN ][cluster.action.shard ] [Hela] sending failed shard for [panda][2], node[iIe-oUNfQ26UcWVBbv9O-Q], [R], s[STARTED], reason [Failed to perform [indices/index/shard/index] on replica, message [RemoteTransportException[[Quicksilver][inet[/192.168.7.216:9300]][indices/index/shard/index/replica]]; nested: IndexFailedEngineException[[panda][2] Index failed for [phonenumber#9-25987-27d8a1ce-98c7-456b-a]]; nested: IOException[No space left on device]; ]]

The index got only around 2G and the exception was very misleading 'cause I've more than enough space left.

I downloaded the ES server, ran as it is (w/ a blank elasticsearch.yml file) and had ulimit set to 64000. Any pointers? Thank a lot!!

2

Its relating to disk space, seems like you ran out of it. Check where the data directory is placed and verify that there is enough space.
On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:

java.io.IOException: No space left on device

3

Thanks!

This is indeed a weird incidence. I have plenty of diskspace(TBs) and even
strace couldn't quite pinpoint exactly which write was causing IOException.
However I found something else intriguing .. if I ran it as it is out of
box(w/ a blank configuration), I'd run into this exception when the index
hit around 2G if I ran it w/ a custom cluster name in elasticsearch.yml, I
could get past that!

This is a single node test run. Is cluster 'elasticsearch' very different
from the rest?

On Tue, Apr 12, 2011 at 2:49 PM, kimchy [via Elasticsearch Users] <
ml-node+2813394-1244121660-390542@n3.nabble.com> wrote:

Its relating to disk space, seems like you ran out of it. Check where the
data directory is placed and verify that there is enough space.

On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:

java.io.IOException: No space left on device

If you reply to this email, your message will be added to the discussion
below:

http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html
To unsubscribe from Failed to perform indices/index/shard/index on replica
Index Shard, click herehttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2805078&code=dW5veGludUBnbWFpbC5jb218MjgwNTA3OHwtMTkyMjgxOTM5NQ==.

4

Nope, the only difference is the internal directory used to store data for the node within the /data location.
On Wednesday, April 13, 2011 at 5:19 AM, unoxinu wrote:

Thanks!

This is indeed a weird incidence. I have plenty of diskspace(TBs) and even strace couldn't quite pinpoint exactly which write was causing IOException. However I found something else intriguing .. if I ran it as it is out of box(w/ a blank configuration), I'd run into this exception when the index hit around 2G if I ran it w/ a custom cluster name in elasticsearch.yml, I could get past that!

This is a single node test run. Is cluster 'elasticsearch' very different from the rest?

On Tue, Apr 12, 2011 at 2:49 PM, kimchy [via Elasticsearch Users] <[hidden email]> wrote:

Its relating to disk space, seems like you ran out of it. Check where the data directory is placed and verify that there is enough space.
On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:

java.io.IOException: No space left on device

If you reply to this email, your message will be added to the discussion below: http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html

View this message in context: Re: Failed to perform indices/index/shard/index on replica Index Shard
Sent from the Elasticsearch Users mailing list archive at Nabble.com.