Getting following exception in one of our Elasticsearch Index,
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[fw][4]
failed recovery]; nested: IllegalArgumentException[Too many documents,
composite IndexReaders cannot exceed 2147483647]; ]]
Please somebody help.
Thanks
--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".
Because, Elasticsearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.
Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".
Because, Elasticsearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.
You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This
is a documented Lucene limit:
"Similarly, Lucene uses a Java int to refer to document numbers, and the
index file format uses an Int32 on-disk to store document numbers. This is
a limitation of both the index file format and the current implementation.
Eventually these should be replaced with either UInt64 values, or better
yet, VInt values which have no limit."
Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".
Because, Elasticsearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.
Conclusion is, we can not store more than 2147483647 records per shard as
of now. The only option is we need to increase the shard count.
Thanks
Prasath Rajan
On Tuesday, October 14, 2014 9:34:33 PM UTC+5:30, Jörg Prante wrote:
You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This
is a documented Lucene limit:
"Similarly, Lucene uses a Java int to refer to document numbers, and the
index file format uses an Int32 on-disk to store document numbers. This is
a limitation of both the index file format and the current implementation.
Eventually these should be replaced with either UInt64 values, or better
yet, VInt values which have no limit."
Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".
Because, Elasticsearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.