# Updating index gives number of documents in the index cannot exceed 2147483519 **URL:** \ **Category:** Elasticsearch\ **Created:** [May 20, 2021, 1:48pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547 "2021-05-20T13:48:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![NLSVTN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nlsvtn/32/84889_2.png) [@NLSVTN](https://discuss.elastic.co/u/NLSVTN)\ **Post date:** [May 20, 2021, 1:48pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/1 "2021-05-20T13:48:33Z")
We are trying to write **42522080** documents into **Elasticsearch** in python using **Hail** : ```auto mt = hl.import_vcf(dataset_path, reference_genome='GRCh' + genome_version, force_bgz=True, min_partitions=500) mt = hl.split_multi_hts(mt.annotate_rows(locus_old=mt.locus, alleles_old=mt.alleles), permit_shuffle=True) ... variant_count = mt.count_rows() logger.info("\n==> exporting {} variants to elasticsearch:".format(variant_count)) row_table = mt.rows().flatten() row_table = row_table.drop(row_table.locus, row_table.alleles) hl.export_elasticsearch(row_table, ...) ``` And we are getting the error: > hail.utils.java.FatalError: EsHadoopException: Could not write all entries for bulk operation [93/1000]. Error sample (first [5] error messages): > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > Bailing out… We are not sure why it reaches **Lucene's** capacity when the actual number of documents that we are trying to create is just **42522080** which we verified (see line with **variant\_count** above). Does it mean that the capacity is applied to inner fields somehow too? The index which has 2 fields less per document was created successfully for us. How could we avoid the issue, what could we try? (Also asked the question on the Hail forum: [Updating index gives number of documents in the index cannot exceed 2147483519 - Hail Query & hailctl - Hail Discussion](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052)) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 20, 2021, 2:25pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/2 "2021-05-20T14:25:02Z")
If you have nested documents each will be stored as a separate document in Lucerne behind the scenes. A single document can therefore result in multiple Lucene documents. Note that the limit is per shard and not per index so you can get around this by increasing the number of primary shards. ---
**Author:** ![NLSVTN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nlsvtn/32/84889_2.png) [@NLSVTN](https://discuss.elastic.co/u/NLSVTN)\ **Post date:** [May 21, 2021, 2:36pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/3 "2021-05-21T14:36:24Z")
Is there a way to determine how many primary shards are needed for a total number of documents? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 21, 2021, 3:40pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/4 "2021-05-21T15:40:56Z")
What is the average number of nested documents per indexed document? ---
**Author:** ![NLSVTN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nlsvtn/32/84889_2.png) [@NLSVTN](https://discuss.elastic.co/u/NLSVTN)\ **Post date:** [May 21, 2021, 3:56pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/5 "2021-05-21T15:56:34Z")
I see there 3 nested fields: one has 7 array fields, another - 26, third - 5. Each array field can have many values. I suppose the answer is 3 then? But `(3 + 1) * 42522080` is below the upper limit. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 21, 2021, 4:25pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/6 "2021-05-21T16:25:09Z")
If a nested field contains an array with 7 JSON objects, that is 7 internal documents in addition to the main one. Your document could therefore require 1+7+26+5=39 internal documents. If you have multiple levels of nesting this naturally increases. ---
**Author:** ![NLSVTN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nlsvtn/32/84889_2.png) [@NLSVTN](https://discuss.elastic.co/u/NLSVTN)\ **Post date:** [May 21, 2021, 4:30pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/7 "2021-05-21T16:30:16Z")
I think I got it. Basically all my 3 nested fields are arrays with the objects that contain the specified number of fields. So, it seems now that I just should know the length of each of the 3 arrays, not the number of fields in the objects that are contained within them. If it is true, then I see why it happens: arrays can have hundreds of elements in them. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 21, 2021, 4:31pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/8 "2021-05-21T16:31:28Z")
Yes, that sounds correct. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 18, 2021, 4:32pm UTC](https://discuss.elastic.co/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/273547/9 "2021-06-18T16:32:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.