Bulk API does not work for huge data!

Elastic Stack Elasticsearch
1

Dear All,

I use bulk Api "$ curl -s -XPOST localhost:9200/_bulk --data-binary
@requests; for a file 1.43GB. But after execution nothing happen!
For small files I do not have any problem.

Thanks to let me know if I have to do something!

Regards,
Ali

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Split your bulk in smaller parts.
For example I'm not injecting bulks with more than 10000 requests.

1gb of data is a way too much!

David

Le 24 févr. 2015 à 23:50, Ali Lotfdar ali.lotfdar58@gmail.com a écrit :

Dear All,

I use bulk Api "$ curl -s -XPOST localhost:9200/_bulk --data-binary @requests; for a file 1.43GB. But after execution nothing happen!
For small files I do not have any problem.

Thanks to let me know if I have to do something!

Regards,
Ali

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ACC2BA47-2936-4EC5-971A-37A94D75D2E2%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Thank you David, so big job for 20000000 requests!

On Tuesday, February 24, 2015 at 6:13:12 PM UTC-5, David Pilato wrote:

Split your bulk in smaller parts.
For example I'm not injecting bulks with more than 10000 requests.

1gb of data is a way too much!

David

Le 24 févr. 2015 à 23:50, Ali Lotfdar <ali.lo...@gmail.com <javascript:>>
a écrit :

Dear All,

I use bulk Api "$ curl -s -XPOST localhost:9200/_bulk --data-binary
@requests; for a file 1.43GB. But after execution nothing happen!
For small files I do not have any problem.

Thanks to let me know if I have to do something!

Regards,
Ali

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

You can increase it by setting this in your cluster:

http.max_content_length: 1500mb

You need a lot of memory to index it though and I have found that big bulk
sizes are actually slower than indexing with smaller bulk sizes.

/JZ

On Wed, Feb 25, 2015 at 2:47 PM, Ali Lotfdar ali.lotfdar58@gmail.com
wrote:

Thank you David, so big job for 20000000 requests!

On Tuesday, February 24, 2015 at 6:13:12 PM UTC-5, David Pilato wrote:

Split your bulk in smaller parts.
For example I'm not injecting bulks with more than 10000 requests.

1gb of data is a way too much!

David

Le 24 févr. 2015 à 23:50, Ali Lotfdar ali.lo...@gmail.com a écrit :

Dear All,

I use bulk Api "$ curl -s -XPOST localhost:9200/_bulk --data-binary
@requests; for a file 1.43GB. But after execution nothing happen!
For small files I do not have any problem.

Thanks to let me know if I have to do something!

Regards,
Ali

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/95afbaff-957f-44f5-8836-e4eef951e05a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2c1536ad-fd78-4c7d-9908-b9bee345553c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAA%2BD3eWrSh5E2RNdEPBzx7v2E-CH0mra7yRRMyFUnAAmdn%2B%2BJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.