ElasticSearch- IndexReaders cannot exceed 2147483647

Getting following exception in one of our ElasticSearch Index,

[Failed to start shard, message [IndexShardGatewayRecoveryException[[fw][4]
failed recovery]; nested: IllegalArgumentException[Too many documents,
composite IndexReaders cannot exceed 2147483647]; ]]

Please somebody help.

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Obvious question: how many documents are you expecting to have in
there? Including the nested ones if you are storing them as separate
documents.

Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

On 14 October 2014 10:25, Prasanth R prasanth.sunrise@gmail.com wrote:

Getting following exception in one of our ElasticSearch Index,

[Failed to start shard, message [IndexShardGatewayRecoveryException[[fw][4]
failed recovery]; nested: IllegalArgumentException[Too many documents,
composite IndexReaders cannot exceed 2147483647]; ]]

Please somebody help.

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEFAe-EfMGvubuVGK%2Bp0BZpAopj7UOVL%2BUE6c18JDAJc_Am-bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

There is no upper limit...

On Tuesday, October 14, 2014 8:02:03 PM UTC+5:30, Alexandre Rafalovitch
wrote:

Obvious question: how many documents are you expecting to have in
there? Including the nested ones if you are storing them as separate
documents.

Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

On 14 October 2014 10:25, Prasanth R <prasanth...@gmail.com <javascript:>>
wrote:

Getting following exception in one of our ElasticSearch Index,

[Failed to start shard, message
[IndexShardGatewayRecoveryException[[fw][4]
failed recovery]; nested: IllegalArgumentException[Too many documents,
composite IndexReaders cannot exceed 2147483647]; ]]

Please somebody help.

Thanks

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3646a7e4-7246-43df-8970-01b776a0720f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On 14 October 2014 10:33, Prasanth R prasanth.sunrise@gmail.com wrote:

There is no upper limit...

Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".

Because, ElasticSearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.

Regards,
Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks for the reply.

My scenario here is,

  1. No nested docs.
  2. I don't have any limit per shard..

I didn't know about internal limit of ES.
On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" arafalov@gmail.com wrote:

On 14 October 2014 10:33, Prasanth R prasanth.sunrise@gmail.com wrote:

There is no upper limit...

Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".

Because, ElasticSearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.

Regards,
Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This
is a documented Lucene limit:

"Similarly, Lucene uses a Java int to refer to document numbers, and the
index file format uses an Int32 on-disk to store document numbers. This is
a limitation of both the index file format and the current implementation.
Eventually these should be replaced with either UInt64 values, or better
yet, VInt values which have no limit."

https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations

Jörg

On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R prasanth.sunrise@gmail.com
wrote:

Thanks for the reply.

My scenario here is,

  1. No nested docs.
  2. I don't have any limit per shard..

I didn't know about internal limit of ES.
On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" arafalov@gmail.com
wrote:

On 14 October 2014 10:33, Prasanth R prasanth.sunrise@gmail.com wrote:

There is no upper limit...

Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".

Because, ElasticSearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.

Regards,
Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFeV%3D6C%2BZWSs85yhZnzU%3D-Eycw4aPZUy4%2BDLYEL_0bXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Dear All,
Thanks for your replies.

Conclusion is, we can not store more than 2147483647 records per shard as
of now. The only option is we need to increase the shard count.

Thanks
Prasath Rajan

On Tuesday, October 14, 2014 9:34:33 PM UTC+5:30, Jörg Prante wrote:

You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This
is a documented Lucene limit:

"Similarly, Lucene uses a Java int to refer to document numbers, and the
index file format uses an Int32 on-disk to store document numbers. This is
a limitation of both the index file format and the current implementation.
Eventually these should be replaced with either UInt64 values, or better
yet, VInt values which have no limit."

https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations

Jörg

On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R <prasanth...@gmail.com
<javascript:>> wrote:

Thanks for the reply.

My scenario here is,

  1. No nested docs.
  2. I don't have any limit per shard..

I didn't know about internal limit of ES.
On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" <araf...@gmail.com
<javascript:>> wrote:

On 14 October 2014 10:33, Prasanth R <prasanth...@gmail.com
<javascript:>> wrote:

There is no upper limit...

Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".

Because, ElasticSearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.

Regards,
Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d85eb58f-2c9d-4fba-88ca-55b64a74be3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.