Multiple errors when transferring data to ES

Elastic Stack Elasticsearch
1

I'm trying to transfer data from S3 to ES using LS.

I'm getting multiple errors like this:

[2015-06-10 19:32:31,184][WARN ][indices.cluster          ] [Water Wizard] [[logstash-2015.01.03][2]] marking and sending shard failed due to [engine failure, reason [merge exception]]
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
	at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:133)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:326)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
	at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:95)
	at org.apache.lucene.store.OutputStreamIndexOutput.writeByte(OutputStreamIndexOutput.java:45)
	at org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:64)
	at org.apache.lucene.store.DataOutput.writeVInt(DataOutput.java:196)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:847)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:647)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:943)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:912)
	at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:112)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:399)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)
	at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)

Any idea what's wrong?

I have 16g for 2 nodes of ES

2

I think this says a lot:

No space left on device

So you run out of space?

3

Crap ... looked at the wrong numbers :disappointed:

Thanks