org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed

Elastic Stack Elasticsearch
1

Hi,
we are doing single large index creation via java program on single elasticsearch node.
after few millions of documents, we faced to this error:

    org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:781) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.elasticsearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:622) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:978) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:182) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:310) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:290) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:213) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
    Caused by: java.io.IOException: Insufficient system resources exist to complete the requested service
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:68) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:286) ~[?:?]
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
	at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
	at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
	at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:416) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:?]
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:123) ~[?:?]
	at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.elasticsearch.common.lucene.store.FilterIndexOutput.writeBytes(FilterIndexOutput.java:59) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:278) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:97) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:36:04]
	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5316) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4825) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4364) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5923) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:100) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:682) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
	Suppressed: java.io.IOException: Insufficient system resources exist to complete the requested service
		at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
		at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:68) ~[?:?]
		at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
		at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
		at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:286) ~[?:?]
		at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
		at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
		at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
		at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:416) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:?]
		at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
		at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
		at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:68) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.elasticsearch.common.lucene.store.FilterIndexOutput.close(FilterIndexOutput.java:39) ~[elasticsearch-7.10.0.jar:7.10.0]
		at org.elasticsearch.index.store.ByteSizeCachingDirectory$2.close(ByteSizeCachingDirectory.java:162) ~[elasticsearch-7.10.0.jar:7.10.0]
		at org.apache.lucene.store.RateLimitedIndexOutput.close(RateLimitedIndexOutput.java:49) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:120) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:36:04]
		at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5316) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4825) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4364) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5923) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:100) ~[elasticsearch-7.10.0.jar:7.10.0]
		at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:682) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
		Suppressed: java.io.IOException: Insufficient system resources exist to complete the requested service
			at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
			at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:68) ~[?:?]
			at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
			at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
			at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:286) ~[?:?]
			at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
			at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
			at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
			at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:416) ~
    [2021-02-13T10:08:00,245][WARN ][o.e.i.e.Engine           ] [WIN-RTGCFDESWSD] [test][1] failed engine [merge failed]
    org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Insufficient system resources exist to complete the requested service
	at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2476) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
    Caused by: java.io.IOException: Insufficient system resources exist to complete the requested service

do you have Any idea what is the problem?

2

Hi @Farzad_Ebrahimi

Looks like this is the deeper issue. Your file system seems to be unable to handle the physical write to disk. The exception isn't super clear but I'd imagine it's some rate limiting/open file count limit or less likely a full disk.

=> I'd investigate the underlying storage here to see why it's not able to cope with the write load.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.