# Is there a plan for lucene to expand max doc size per index? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 2, 2016, 9:39am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690 "2016-11-02T09:39:54Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![makeyang](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@makeyang](https://discuss.elastic.co/u/makeyang)\ **Post date:** [November 2, 2016, 9:39am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/1 "2016-11-02T09:39:54Z")
init design is 2^32. but as the data is growing fast and growing big, is there a plan to make it 2^64 or something bigger than current value ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 2, 2016, 9:41am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/2 "2016-11-02T09:41:38Z")
It's actually _a per shard_ limit, so you can have lots more than 2^32 per index 🙂 But, @mikemccand, any thoughts on this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 2, 2016, 9:58am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/3 "2016-11-02T09:58:24Z")
The limit is as Mark pointed out per shard, and not per index. As queries are executed in parallel across shards, but are single threaded for each shard, query performance tend to depend on the shard size. Having more than 2^32 documents in a shard would probably result in very large shards, which can also cause problems during recovery, when shards need to be moved around in the cluster. How many documents do you currently have in the your shards? How large are your shards? ---
**Author:** ![makeyang](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@makeyang](https://discuss.elastic.co/u/makeyang)\ **Post date:** [November 2, 2016, 10:15am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/4 "2016-11-02T10:15:19Z")
the total doc is about 14,000,000,000 actually I shard index based on certain key and some shard will over 2^32. in this case, more shard would mitigate the problem rather than resolve it. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 2, 2016, 10:28pm UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/5 "2016-11-02T22:28:51Z")
It sounds a bit like you are working against ES here, why not let it shard things itself? ---
**Author:** ![makeyang](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@makeyang](https://discuss.elastic.co/u/makeyang)\ **Post date:** [November 3, 2016, 2:58am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/6 "2016-11-03T02:58:25Z")
it's bounded to businuss logic, man ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 3, 2016, 5:27am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/7 "2016-11-03T05:27:29Z")
That doesn't really it though. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 3, 2016, 9:29am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/8 "2016-11-03T09:29:30Z")
> [@makeyang](#): > > actually I shard index based on certain key If you have a key to break down a single index into shards that become too large with regard to document count, consider settiing up the key as an index name and provide an entire index per key. Small indices may get small shard count, while large indices may get higher shard count. ---
**Author:** ![makeyang](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@makeyang](https://discuss.elastic.co/u/makeyang)\ **Post date:** [November 3, 2016, 9:41am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/9 "2016-11-03T09:41:35Z")
sure. there are solutions to resolve my issue. but I wonder why expand max doc size for lucene isn't the option for u guys? ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 3, 2016, 2:09pm UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/10 "2016-11-03T14:09:17Z")
The status of Lucene as of Elasticsearch 2.x is: [http://lucene.apache.org/core/5\_2\_0/core/org/apache/lucene/codecs/lucene50/package-summary.html#Limitations](http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/codecs/lucene50/package-summary.html#Limitations) > Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either UInt64 values, or better yet, VInt values which have no limit. For ES 5 which uses Lucene 6.2, the API did not change much [http://lucene.apache.org/core/6\_2\_0/core/org/apache/lucene/index/SegmentInfo.html](http://lucene.apache.org/core/6_2_0/core/org/apache/lucene/index/SegmentInfo.html) There is `maxDoc()` still returning a Java int which is to be interpreted as a `uint32`. You could change the Lucene 6.2 source code, rewrite `SegmentInfo`, and recompile a custom Lucene togetheer with a recompiled ES 5 on top of it. The Lucene developers noted they intend to encode the document numbers no longer as int32 but as a VInt so I think there is no reason why not to do this. ---
**Author:** ![makeyang](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@makeyang](https://discuss.elastic.co/u/makeyang)\ **Post date:** [November 4, 2016, 2:39am UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/11 "2016-11-04T02:39:19Z")
thanks man. this is exactly what I want. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:06pm UTC](https://discuss.elastic.co/t/is-there-a-plan-for-lucene-to-expand-max-doc-size-per-index/64690/12 "2017-07-05T22:06:42Z")