# Failed to perform indices/index/shard/index on replica Index Shard
**URL:** https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210
**Category:** Elasticsearch
**Created:** [April 11, 2011, 3:06am UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210 "2011-04-11T03:06:53Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@unoxinu](https://discuss.elastic.co/u/unoxinu)
#### Post date: [April 11, 2011, 3:06am UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210/1 "2011-04-11T03:06:53Z")
Hi all,
I looked around and couldn't find an answer for this weird exception I've been getting half-way through the indexing of about 1M documents(each one has an ID field and a phonenumbers field):
Failed to perform indices/index/shard/index on replica Index Shard [panda][2]org.elasticsearch.transport.RemoteTransportException: [Quicksilver][inet[/192.168.7.216:9300]][indices/index/shard/index/repl
ica]Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: [panda][2] Index failed for [phonenumber#9-25987-27d8a1
ce-98c7-456b-a]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:329)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:284)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnReplica(TransportIndexAction.java:227) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHan
dler.messageReceived(TransportShardReplicationOperationAction.java:202) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHan
dler.messageReceived(TransportShardReplicationOperationAction.java:191)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.flushBuffer(SimpleFSDirectory.java:187)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:77)
at org.elasticsearch.index.store.support.AbstractStore$StoreIndexOutput.writeBytes(AbstractStore.java:350)
```
at org.apache.lucene.store.IndexOutput.writeBytes(IndexOutput.java:43)
at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:66)
at org.apache.lucene.index.FieldsWriter.flushDocument(FieldsWriter.java:129)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:149)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:802)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1998)
at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:391)
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:322)
... 8 more
```
[2011-04-10 20:27:33,959][WARN][cluster.action.shard] [Hela] sending failed shard for [panda][2], node[iIe-oUNfQ26UcWVBbv9O-Q], [R], s[STARTED], reason [Failed to perform [indices/index/shard/index] on replica, message [RemoteTransportException[[Quicksilver][inet[/192.168.7.216:9300]][indices/index/shard/index/replica]]; nested: IndexFailedEngineException[[panda][2] Index failed for [phonenumber#9-25987-27d8a1ce-98c7-456b-a]]; nested: IOException[No space left on device]; ]]
The index got only around 2G and the exception was very misleading 'cause I've more than enough space left.
I downloaded the ES server, ran as it is (w/ a blank elasticsearch.yml file) and had ulimit set to 64000. Any pointers? Thank a lot!!
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 12, 2011, 9:49pm UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210/2 "2011-04-12T21:49:13Z")
Its relating to disk space, seems like you ran out of it. Check where the data directory is placed and verify that there is enough space.
On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:
> java.io.IOException: No space left on device
---
### Author:  [@unoxinu](https://discuss.elastic.co/u/unoxinu)
#### Post date: [April 13, 2011, 2:19am UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210/3 "2011-04-13T02:19:52Z")
Thanks!
This is indeed a weird incidence. I have plenty of diskspace(TBs) and even
strace couldn't quite pinpoint exactly which write was causing IOException.
However I found something else intriguing .. if I ran it as it is out of
box(w/ a blank configuration), I'd run into this exception when the index
hit around 2G if I ran it w/ a custom cluster name in elasticsearch.yml, I
could get past that!
This is a single node test run. Is cluster 'elasticsearch' very different
from the rest?
On Tue, Apr 12, 2011 at 2:49 PM, kimchy [via Elasticsearch Users] \<
[ml-node+2813394-1244121660-390542@n3.nabble.com](mailto:ml-node+2813394-1244121660-390542@n3.nabble.com)\> wrote:
> Its relating to disk space, seems like you ran out of it. Check where the
> data directory is placed and verify that there is enough space.
>
> On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:
>
> java.io.IOException: No space left on device
>
> * * *
>
> If you reply to this email, your message will be added to the discussion
> below:
>
> [http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html](http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html)
> To unsubscribe from Failed to perform indices/index/shard/index on replica
> Index Shard, click here[http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe\_by\_code&node=2805078&code=dW5veGludUBnbWFpbC5jb218MjgwNTA3OHwtMTkyMjgxOTM5NQ==](http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2805078&code=dW5veGludUBnbWFpbC5jb218MjgwNTA3OHwtMTkyMjgxOTM5NQ==).
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 15, 2011, 10:12pm UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210/4 "2011-04-15T22:12:51Z")
Nope, the only difference is the internal directory used to store data for the node within the /data location.
On Wednesday, April 13, 2011 at 5:19 AM, unoxinu wrote:
> Thanks!
>
> This is indeed a weird incidence. I have plenty of diskspace(TBs) and even strace couldn't quite pinpoint exactly which write was causing IOException. However I found something else intriguing .. if I ran it as it is out of box(w/ a blank configuration), I'd run into this exception when the index hit around 2G if I ran it w/ a custom cluster name in elasticsearch.yml, I could get past that!
>
> This is a single node test run. Is cluster 'elasticsearch' very different from the rest?
>
> On Tue, Apr 12, 2011 at 2:49 PM, kimchy [via Elasticsearch Users] \<[hidden email]\> wrote:
>
> > Its relating to disk space, seems like you ran out of it. Check where the data directory is placed and verify that there is enough space.
> > On Monday, April 11, 2011 at 6:06 AM, unoxinu wrote:
> >
> > > java.io.IOException: No space left on device
> >
> > If you reply to this email, your message will be added to the discussion below: [http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html](http://elasticsearch-users.115913.n3.nabble.com/Failed-to-perform-indices-index-shard-index-on-replica-Index-Shard-tp2805078p2813394.html)
>
> View this message in context: Re: Failed to perform indices/index/shard/index on replica Index Shard
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:08am UTC](https://discuss.elastic.co/t/failed-to-perform-indices-index-shard-index-on-replica-index-shard/4210/5 "2017-07-06T04:08:15Z")