Limitation of 2,147,483,647 terms per segment index in Lucene

Elastic Stack Elasticsearch
1

Is there a way to recover a segment index of a shard that has exceeded
Lucene's 2^31 limit?

Thanks,
-peter

[2014-06-30 10:53:02,187][WARN ][indices.cluster ] [Patriots]
[vos][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[vos][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: Too many documents,
composite IndexReaders cannot exceed 2147483647
at
org.apache.lucene.index.BaseCompositeReader.(BaseCompositeReader.java:77)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:369)
at
org.apache.lucene.index.StandardDirectoryReader.(StandardDirectoryReader.java:43)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:115)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1364)
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:291)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:709)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:204)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
... 3 more
[2014-06-30 10:53:02,213][WARN ][cluster.action.shard ] [Patriots]
[vos][0] sending failed shard for [vos][0], node[bS9Lp_a9QZOjiab23Ztk4A],
[P], s[INITIALIZING], indexUUID [a0_HJrlgQq-UNCwL2QiVbg], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[vos][0] failed
recovery]; nested: IllegalArgumentException[Too many documents, composite
IndexReaders cannot exceed 2147483647]; ]]
[2014-06-30 10:53:02,213][WARN ][cluster.action.shard ] [Patriots]
[vos][0] received shard failed for [vos][0], node[bS9Lp_a9QZOjiab23Ztk4A],
[P], s[INITIALIZING], indexUUID [a0_HJrlgQq-UNCwL2QiVbg], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[vos][0] failed
recovery]; nested: IllegalArgumentException[Too many documents, composite
IndexReaders cannot exceed 2147483647]; ]]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5e2a408-f1e5-469a-9bcc-679f399dbe9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Peter, thanks so much for raising this. This looks aweful! I think we
should move this into an issue on [1] (please feel free to create one) IMO
we should aim to name the issue in a way to prevent this from happening
altogether. Along the lines we should help you to recover but I don't know
how tricky it will be. Lets start with the issue!!

simon

[1] Issues · elastic/elasticsearch · GitHub

On Monday, June 30, 2014 5:49:32 PM UTC+2, Peter Portante wrote:

Is there a way to recover a segment index of a shard that has exceeded
Lucene's 2^31 limit?

Thanks,
-peter

[2014-06-30 10:53:02,187][WARN ][indices.cluster ] [Patriots]
[vos][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[vos][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: Too many documents,
composite IndexReaders cannot exceed 2147483647
at
org.apache.lucene.index.BaseCompositeReader.(BaseCompositeReader.java:77)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:369)
at
org.apache.lucene.index.StandardDirectoryReader.(StandardDirectoryReader.java:43)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:115)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1364)
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:291)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:709)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:204)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
... 3 more
[2014-06-30 10:53:02,213][WARN ][cluster.action.shard ] [Patriots]
[vos][0] sending failed shard for [vos][0], node[bS9Lp_a9QZOjiab23Ztk4A],
[P], s[INITIALIZING], indexUUID [a0_HJrlgQq-UNCwL2QiVbg], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[vos][0] failed
recovery]; nested: IllegalArgumentException[Too many documents, composite
IndexReaders cannot exceed 2147483647]; ]]
[2014-06-30 10:53:02,213][WARN ][cluster.action.shard ] [Patriots]
[vos][0] received shard failed for [vos][0], node[bS9Lp_a9QZOjiab23Ztk4A],
[P], s[INITIALIZING], indexUUID [a0_HJrlgQq-UNCwL2QiVbg], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[vos][0] failed
recovery]; nested: IllegalArgumentException[Too many documents, composite
IndexReaders cannot exceed 2147483647]; ]]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01253247-c16b-44cb-aa61-d02bc20f44eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.