# Bulk index request dataloss **URL:** \ **Category:** Elasticsearch\ **Created:** [April 7, 2015, 9:16am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127 "2015-04-07T09:16:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![am87](https://avatars.discourse-cdn.com/v4/letter/a/e36b37/32.png) [@am87](https://discuss.elastic.co/u/am87)\ **Post date:** [April 7, 2015, 9:16am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/1 "2015-04-07T09:16:05Z")
Hey everyone, I've been trying to maximise my indexing rate. I'm indexing around a million documents, using 4 threads. Each thread is indexing at 2500 documents per bulk request, so that's 10000 at a time. I've been playing around with these parameters and found that if I go higher and higher, I eventually start getting data-loss. The index ends up with less than 1,000,000 documents every time. There are no error in the logs, so I'm not sure what's causing this. Taking these parameters down a notch fixes this problem. Has anyone seen this issue before? Is there anything that can be done about it Thankyou -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 7, 2015, 10:10am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/2 "2015-04-07T10:10:22Z")
Do you evaluate the bulk request responses? Jörg On Tue, Apr 7, 2015 at 11:16 AM, mzrth\_7810 [afrazmamoon@gmail.com](mailto:afrazmamoon@gmail.com) wrote: > Hey everyone, > > I've been trying to maximise my indexing rate. I'm indexing around a > million documents, using 4 threads. Each thread is indexing at 2500 > documents per bulk request, so that's 10000 at a time. > > I've been playing around with these parameters and found that if I go > higher and higher, I eventually start getting data-loss. The index ends up > with less than 1,000,000 documents every time. There are no error in the > logs, so I'm not sure what's causing this. > > Taking these parameters down a notch fixes this problem. > > Has anyone seen this issue before? > Is there anything that can be done about it > > Thankyou > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGk0XDn7KozQrWmHjW-zW89edQcDNhcxnn57JZDfqYuaw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGk0XDn7KozQrWmHjW-zW89edQcDNhcxnn57JZDfqYuaw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![am87](https://avatars.discourse-cdn.com/v4/letter/a/e36b37/32.png) [@am87](https://discuss.elastic.co/u/am87)\ **Post date:** [April 23, 2015, 10:20am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/3 "2015-04-23T10:20:24Z")
Turns out it was because the bulk thread pool queue size was too small, any new requests were being rejected. Is it common to set threadpool.bulk.queue\_size to something like 1000 ? On Tuesday, 7 April 2015 11:10:33 UTC+1, Jörg Prante wrote: > Do you evaluate the bulk request responses? > > Jörg > > On Tue, Apr 7, 2015 at 11:16 AM, mzrth\_7810 \<[afraz...@gmail.com](mailto:afraz...@gmail.com) > \\> wrote: > > > Hey everyone, > > > > I've been trying to maximise my indexing rate. I'm indexing around a > > million documents, using 4 threads. Each thread is indexing at 2500 > > documents per bulk request, so that's 10000 at a time. > > > > I've been playing around with these parameters and found that if I go > > higher and higher, I eventually start getting data-loss. The index ends up > > with less than 1,000,000 documents every time. There are no error in the > > logs, so I'm not sure what's causing this. > > > > Taking these parameters down a notch fixes this problem. > > > > Has anyone seen this issue before? > > Is there anything that can be done about it > > > > Thankyou > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 23, 2015, 10:53am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/4 "2015-04-23T10:53:49Z")
It would mean that you are going to accumulate up to 1000 requests of 2500 docs at a time in memory. That could be a lot. You need to monitor that. That’s a lot of objects that might be GCed at some point. If your bulk request is rejected, why not trying to slow down injection rate instead of filling the memory? You could also think of setting replicas to 0 before bulk and the reactivate to 1 after injection. Having SSD drives can also help but may be you have already that? My 2 cents -- David Pilato - Developer | Evangelist > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co/)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 23 avr. 2015 à 12:20, mzrth\_7810 [afrazmamoon@gmail.com](mailto:afrazmamoon@gmail.com) a écrit : > > Turns out it was because the bulk thread pool queue size was too small, any new requests were being rejected. > > Is it common to set threadpool.bulk.queue\_size to something like 1000 ? > > On Tuesday, 7 April 2015 11:10:33 UTC+1, Jörg Prante wrote: > Do you evaluate the bulk request responses? > > Jörg > > On Tue, Apr 7, 2015 at 11:16 AM, mzrth\_7810 \<[afraz...@gmail.com](mailto:afraz...@gmail.com) \\> wrote: > Hey everyone, > > I've been trying to maximise my indexing rate. I'm indexing around a million documents, using 4 threads. Each thread is indexing at 2500 documents per bulk request, so that's 10000 at a time. > > I've been playing around with these parameters and found that if I go higher and higher, I eventually start getting data-loss. The index ends up with less than 1,000,000 documents every time. There are no error in the logs, so I'm not sure what's causing this. > > Taking these parameters down a notch fixes this problem. > > Has anyone seen this issue before? > Is there anything that can be done about it > > Thankyou > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/E4EC9E42-9DCA-410A-846F-1562B256D8DC%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/E4EC9E42-9DCA-410A-846F-1562B256D8DC%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 23, 2015, 11:15am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/5 "2015-04-23T11:15:56Z")
With the JDBC plugin, you should slightly increase the requests per bulk request ("maxbulkactions") in order to keep your concurrent bulk requests low enough to get handled by ES. The ES bulk thread pool default setting is ok. Please avoid a change. Jörg On Thu, Apr 23, 2015 at 12:20 PM, mzrth\_7810 [afrazmamoon@gmail.com](mailto:afrazmamoon@gmail.com) wrote: > Turns out it was because the bulk thread pool queue size was too small, > any new requests were being rejected. > > Is it common to set threadpool.bulk.queue\_size to something like 1000 ? > > On Tuesday, 7 April 2015 11:10:33 UTC+1, Jörg Prante wrote: > > > Do you evaluate the bulk request responses? > > > > Jörg > > > > On Tue, Apr 7, 2015 at 11:16 AM, mzrth\_7810 [afraz...@gmail.com](mailto:afraz...@gmail.com) wrote: > > > > > Hey everyone, > > > > > > I've been trying to maximise my indexing rate. I'm indexing around a > > > million documents, using 4 threads. Each thread is indexing at 2500 > > > documents per bulk request, so that's 10000 at a time. > > > > > > I've been playing around with these parameters and found that if I go > > > higher and higher, I eventually start getting data-loss. The index ends up > > > with less than 1,000,000 documents every time. There are no error in the > > > logs, so I'm not sure what's causing this. > > > > > > Taking these parameters down a notch fixes this problem. > > > > > > Has anyone seen this issue before? > > > Is there anything that can be done about it > > > > > > Thankyou > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHx-fKLQLLqtZNM8mnupf\_8n%3DMjviyx6NaqqAB7eHJFTA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHx-fKLQLLqtZNM8mnupf_8n%3DMjviyx6NaqqAB7eHJFTA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![am87](https://avatars.discourse-cdn.com/v4/letter/a/e36b37/32.png) [@am87](https://discuss.elastic.co/u/am87)\ **Post date:** [April 29, 2015, 10:13am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/6 "2015-04-29T10:13:07Z")
@David, that makes sense. We have SSDs and replicas are already set to 0 while bulk indexing. @Jörg, we haven't changed the "threadpool.bulk.size" because according to the docs that's directly related to the number of processors available. However "threadpool.bulk.queue\_size" has been modified. I'm slowly tuning it down to find a sweetspot, but the default seems a but too low. On Thursday, 23 April 2015 12:16:03 UTC+1, Jörg Prante wrote: > With the JDBC plugin, you should slightly increase the requests per bulk > request ("maxbulkactions") in order to keep your concurrent bulk requests > low enough to get handled by ES. > > The ES bulk thread pool default setting is ok. Please avoid a change. > > Jörg > > On Thu, Apr 23, 2015 at 12:20 PM, mzrth\_7810 \<[afraz...@gmail.com](mailto:afraz...@gmail.com) > \\> wrote: > > > Turns out it was because the bulk thread pool queue size was too small, > > any new requests were being rejected. > > > > Is it common to set threadpool.bulk.queue\_size to something like 1000 ? > > > > On Tuesday, 7 April 2015 11:10:33 UTC+1, Jörg Prante wrote: > > > > > Do you evaluate the bulk request responses? > > > > > > Jörg > > > > > > On Tue, Apr 7, 2015 at 11:16 AM, mzrth\_7810 [afraz...@gmail.com](mailto:afraz...@gmail.com) wrote: > > > > > > > Hey everyone, > > > > > > > > I've been trying to maximise my indexing rate. I'm indexing around a > > > > million documents, using 4 threads. Each thread is indexing at 2500 > > > > documents per bulk request, so that's 10000 at a time. > > > > > > > > I've been playing around with these parameters and found that if I go > > > > higher and higher, I eventually start getting data-loss. The index ends up > > > > with less than 1,000,000 documents every time. There are no error in the > > > > logs, so I'm not sure what's causing this. > > > > > > > > Taking these parameters down a notch fixes this problem. > > > > > > > > Has anyone seen this issue before? > > > > Is there anything that can be done about it > > > > > > > > Thankyou > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c531c868-19ce-4d2e-9744-d043463ed084%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c531c868-19ce-4d2e-9744-d043463ed084%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:17am UTC](https://discuss.elastic.co/t/bulk-index-request-dataloss/23127/7 "2017-07-06T00:17:07Z")