# Indexing crashing due to large dataset **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [February 7, 2024, 8:48am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715 "2024-02-07T08:48:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![juliebryan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juliebryan/32/124970_2.png) [@juliebryan](https://discuss.elastic.co/u/juliebryan)\ **Post date:** [February 7, 2024, 8:48am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/1 "2024-02-07T08:48:59Z")
I have an Elasticsearch instance running on a docker. I am rootless user. I am also a beginner with Elasticsearch. My data size is 20GB, and they are medical documents. I have 80GB of RAM available. Here is how I configure my Elasticsearch. ```auto Settings = { "settings": { "number_of_shards": 600, "number_of_replicas": 0, "analysis": { "analyzer": { "custom_analyzer": { "type":"custom", "tokenizer": "whitespace", "filter": ["lowercase","kstem"] } }, "filter":{ "kstem": { "type":"kstem" } } } }, "mappings": { "properties": { "ArticleTitle": { "type": "text", "analyzer": "custom_analyzer" }, "AbstractText":{ "type":"text", "analyzer":"custom_analyzer" }, "PMID":{ "type": "keyword", "index": "false" } } } } ``` My index crashes after indexing 12-15 GB of data. Please let me know if there is a more efficient way to configure my index given the constraints. I have tried increasing the heap size by custom JVM\_options file. However it does not work. I configured the heap size to be 35GB. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2024, 8:59am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/2 "2024-02-07T08:59:54Z")
Which version of Elasticsearch are you using? Why have you set the number of primary shards to 600 if your data size is only 20GB? > [@juliebryan](#): > > My index crashes after indexing 12-15 GB of data. How are you indexing data? What bulk size are you using? What is the average raw document size? > [@juliebryan](#): > > I configured the heap size to be 35GB. You should not set the heap beyond around 30GB as you want to benefit from compressed pointers. ---
**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\ **Post date:** [February 7, 2024, 9:03am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/3 "2024-02-07T09:03:07Z")
> [@juliebryan](#): > > I have an Elasticsearch instance running on a docker. I am rootless user. I am also a beginner with Elasticsearch. My data size is 20GB, and they are medical documents. I have 80GB of RAM available. Here is how I configure my Elasticsearch. > > ```auto > Settings = { > "settings": { > "number_of_shards": 600, > "number_of_replicas": 0, > "analysis": { > "analyzer": { > "custom_analyzer": { > "type":"custom", > "tokenizer": "whitespace", > "filter": ["lowercase","kstem"] > } > }, > "filter":{ > "kstem": { > "type":"kstem" > } > } > } > }, > "mappings": { > "properties": { > "ArticleTitle": { > "type": "text", > "analyzer": "custom_analyzer" > }, > "AbstractText":{ > "type":"text", > "analyzer":"custom_analyzer" > }, > "PMID":{ > "type": "keyword", > "index": "false" > } > } > } > } > > ``` > > My index crashes after indexing 12-15 GB of data. Please let me know if there is a more efficient way to configure my index given the constraints. I have tried increasing the heap size by custom JVM\_options file. However it does not work. I configured the heap size to be 35GB. Hi, A common rule of thumb is to aim for shard sizes between 10GB and 50GB, so you might want to start with 1 or 2 shards and adjust based on your performance. As Christian as said, the heap size to 35GB might not be optimal. Regards ---
**Author:** ![juliebryan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juliebryan/32/124970_2.png) [@juliebryan](https://discuss.elastic.co/u/juliebryan)\ **Post date:** [February 7, 2024, 3:20pm UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/4 "2024-02-07T15:20:49Z")
Since I could find a good guide on it, I selected an older version - 7.6.0. I am using the Helpers bulk API for indexing. My data is a pandas data frame of size 780MB. There are 5 such data frame which I pass to the API. I read in the documentation that a good measure is 20 shards/GB. I planned to use 30GB of heap, and thus the 600 shards. ---
**Author:** ![juliebryan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juliebryan/32/124970_2.png) [@juliebryan](https://discuss.elastic.co/u/juliebryan)\ **Post date:** [February 7, 2024, 3:28pm UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/5 "2024-02-07T15:28:08Z")
I will try indexing with reduced number of shards and heap size. I'll post any error I encounter here. I think I might be understanding the error incorrectly. Your help will be massively helpful! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2024, 4:05pm UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/6 "2024-02-07T16:05:27Z")
> [@juliebryan](#): > > My data is a pandas data frame of size 780MB. There are 5 such data frame which I pass to the API. It sounds like the size of your bulk requests may be very large. If this is the case I would recommend you shrink them. A good size to aim for is 5MB to 10MB. > [@juliebryan](#): > > I read in the documentation that a good measure is 20 shards/GB. I planned to use 30GB of heap, and thus the 600 shards. You have misread this. The recommendation is to have an average shard size between 20GB and 50GB. This is what you should primarily aim for. The shard count to heap size is a **maximum** , not a recommended value. This came about as I saw a lot of clusters where users craeted far to many small shards and ended up with problems because of that. This does not apply to your use case, so you should likely go with 1 or 2 primary shards. ---
**Author:** ![juliebryan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juliebryan/32/124970_2.png) [@juliebryan](https://discuss.elastic.co/u/juliebryan)\ **Post date:** [February 22, 2024, 5:47am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/7 "2024-02-22T05:47:40Z")
Thank you so much. I just indexed all my data successfully! It took a long time because I was occupied in other things. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 22, 2024, 6:40am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/8 "2024-02-22T06:40:58Z")
I hope you changed the primary shard count to something more reasonable. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 21, 2024, 6:41am UTC](https://discuss.elastic.co/t/indexing-crashing-due-to-large-dataset/352715/9 "2024-03-21T06:41:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.