We have been fighting with this issue for several months now. We've had it
occur in 0.90.6 - 0.90.13 and now in 1.1, too. It went away for a while
after we upgraded to 1.1 so we thought we had beaten it, but it cropped up
again last night. Any thoughts or help appreciated.
The issue seems to occur most often during or immediately after we run an
index. It always happens on the exact same index, too. One, maybe two,
shards will fail and queries against that index will fail from then on.
We've poured through the code that indexes this data and can't see
anything. We also don't know why client code would be able to kill a
server. All data nodes in the cluster are CentOS 6.5 servers running Java
1.7.0u51. All non-data clients are using a java client library. There is 1
non-data node for the client that loads data and 2 more non-data nodes
serving the front-end. Let me know if there is any other information you
think would be pertinent. Thanks for your help.
[2014-04-23 15:43:14,141][INFO ][cluster.metadata ]
[servicebus-es1.example.com] [product] update_mapping [product] (dynamic)
[2014-04-23 15:46:00,040][INFO ][cluster.metadata ]
[servicebus-es1.example.com] [product] update_mapping [product] (dynamic)
[2014-04-23 15:47:26,884][WARN ][transport ]
[servicebus-es1.example.com] Received response for a request that has timed
out, sent [31301ms] ago, timed out [1300ms] ago, action
[discovery/zen/fd/ping], node
[[servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true,
data=false, local=false}], id [160544]
[2014-04-23 16:30:38,734][INFO ][cluster.service ]
[servicebus-es1.example.com] removed
{[servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true,
data=false, local=false},}, reason:
zen-disco-node_left([servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true,
data=false, local=false})
[2014-04-23 16:30:49,756][INFO ][cluster.service ]
[servicebus-es1.example.com] added
{[servicebus-PROD-tomcatapps3.example.com][ixCfue8YRMq3XhOOcdlmEg][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true,
data=false, local=false},}, reason: zen-disco-receive(join from
node[[servicebus-PROD-tomcatapps3.example.com][ixCfue8YRMq3XhOOcdlmEg][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true,
data=false, local=false}])
[2014-04-23 16:35:51,054][WARN ][index.translog ]
[servicebus-es1.example.com] [product][4] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:35:54,599][WARN ][index.translog ]
[servicebus-es1.example.com] [product][1] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:02,389][WARN ][index.translog ]
[servicebus-es1.example.com] [product][1] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:06,758][WARN ][index.translog ]
[servicebus-es1.example.com] [product][4] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:11,683][WARN ][index.translog ]
[servicebus-es1.example.com] [product][1] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:14,900][WARN ][index.translog ]
[servicebus-es1.example.com] [product][4] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:23,382][WARN ][index.translog ]
[servicebus-es1.example.com] [product][4] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:24,811][WARN ][index.translog ]
[servicebus-es1.example.com] [product][1] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:28,577][WARN ][index.translog ]
[servicebus-es1.example.com] [product][4] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-04-23 16:36:41,739][WARN ][index.translog ]
[servicebus-es1.example.com] [product][1] failed to flush shard on translog
threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1]
Flush failed
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826)
at
org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at
org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader
is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102)
at
org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56)
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122)
at
org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526)
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at
org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/533015d6-3757-40c1-9eac-a3ef61a4f46b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.