# Bulk API does not work for huge data! **URL:** \ **Category:** Elasticsearch\ **Created:** [February 24, 2015, 10:50pm UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366 "2015-02-24T22:50:08Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ali\_Lotfdar](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@Ali\_Lotfdar](https://discuss.elastic.co/u/Ali_Lotfdar)\ **Post date:** [February 24, 2015, 10:50pm UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366/1 "2015-02-24T22:50:08Z")
Dear All, I use bulk Api "$ curl -s -XPOST localhost:9200/\_bulk --data-binary @requests; for a file 1.43GB. But after execution nothing happen! For small files I do not have any problem. Thanks to let me know if I have to do something! Regards, Ali -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 24, 2015, 11:12pm UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366/2 "2015-02-24T23:12:50Z")
Split your bulk in smaller parts. For example I'm not injecting bulks with more than 10000 requests. 1gb of data is a way too much! David > Le 24 févr. 2015 à 23:50, Ali Lotfdar [ali.lotfdar58@gmail.com](mailto:ali.lotfdar58@gmail.com) a écrit : > > Dear All, > > I use bulk Api "$ curl -s -XPOST localhost:9200/\_bulk --data-binary @requests; for a file 1.43GB. But after execution nothing happen! > For small files I do not have any problem. > > Thanks to let me know if I have to do something! > > ## Regards, Ali > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ACC2BA47-2936-4EC5-971A-37A94D75D2E2%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/ACC2BA47-2936-4EC5-971A-37A94D75D2E2%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ali\_Lotfdar](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@Ali\_Lotfdar](https://discuss.elastic.co/u/Ali_Lotfdar)\ **Post date:** [February 25, 2015, 1:47pm UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366/3 "2015-02-25T13:47:01Z")
Thank you David, so big job for 20000000 requests! On Tuesday, February 24, 2015 at 6:13:12 PM UTC-5, David Pilato wrote: > Split your bulk in smaller parts. > For example I'm not injecting bulks with more than 10000 requests. > > 1gb of data is a way too much! > > David > > Le 24 févr. 2015 à 23:50, Ali Lotfdar \<[ali.lo...@gmail.com](mailto:ali.lo...@gmail.com) \\> > a écrit : > > Dear All, > > I use bulk Api "$ curl -s -XPOST localhost:9200/\_bulk --data-binary > @requests; for a file 1.43GB. But after execution nothing happen! > For small files I do not have any problem. > > Thanks to let me know if I have to do something! > > Regards, > Ali > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![JZ1](https://avatars.discourse-cdn.com/v4/letter/j/958977/32.png) [@JZ1](https://discuss.elastic.co/u/JZ1)\ **Post date:** [February 25, 2015, 4:28pm UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366/4 "2015-02-25T16:28:26Z")
You can increase it by setting this in your cluster: http.max\_content\_length: 1500mb You need a lot of memory to index it though and I have found that big bulk sizes are actually slower than indexing with smaller bulk sizes. /JZ On Wed, Feb 25, 2015 at 2:47 PM, Ali Lotfdar [ali.lotfdar58@gmail.com](mailto:ali.lotfdar58@gmail.com) wrote: > Thank you David, so big job for 20000000 requests! > > On Tuesday, February 24, 2015 at 6:13:12 PM UTC-5, David Pilato wrote: > > > Split your bulk in smaller parts. > > For example I'm not injecting bulks with more than 10000 requests. > > > > 1gb of data is a way too much! > > > > David > > > > Le 24 févr. 2015 à 23:50, Ali Lotfdar [ali.lo...@gmail.com](mailto:ali.lo...@gmail.com) a écrit : > > > > Dear All, > > > > I use bulk Api "$ curl -s -XPOST localhost:9200/\_bulk --data-binary > > @requests; for a file 1.43GB. But after execution nothing happen! > > For small files I do not have any problem. > > > > Thanks to let me know if I have to do something! > > > > Regards, > > Ali > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a% > > [40googlegroups.com](http://40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAA%2BD3eWrSh5E2RNdEPBzx7v2E-CH0mra7yRRMyFUnAAmdn%2B%2BJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAA%2BD3eWrSh5E2RNdEPBzx7v2E-CH0mra7yRRMyFUnAAmdn%2B%2BJA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:30am UTC](https://discuss.elastic.co/t/bulk-api-does-not-work-for-huge-data/22366/5 "2017-07-06T00:30:09Z")