# Curl timeout during bulk insert **URL:** \ **Category:** Elasticsearch\ **Created:** [March 28, 2013, 12:54pm UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355 "2013-03-28T12:54:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Bastiaan\_Zijlema](https://avatars.discourse-cdn.com/v4/letter/b/977dab/32.png) [@Bastiaan\_Zijlema](https://discuss.elastic.co/u/Bastiaan_Zijlema)\ **Post date:** [March 28, 2013, 12:54pm UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/1 "2013-03-28T12:54:51Z")
Hello everybody, We have an ES index with around 1 billion documents. The index is constantly updated and expanded by 16 spiders. The spiders send bulk inserts with curl (http api), varying between 1 and 10,000 documents, depending on the data found by the spiders. Everything is running fine, the inserts take a few seconds at most. However, once in a while the curl request will time out after 2 minutes. I'm trying to find out why this is happening, but I cannot find anything useful in the logs. I only see some warnings from the garbage collector, but not at the times the timeouts happen. So I have two questions: - how can I find out why the timeouts are happening (log settings)? - Is this a bad thing, or should I just accept this will happen and increase the curl timeout settings? Our setup: - 4 nodes - 4 cores per node - 24GB Ram, 16GB heap size per node - Java 1.7 - ES 0.20.5 ES settings: - 12 shards - 1 replica Thanks in advance. Regards, Bastiaan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [March 28, 2013, 4:39pm UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/2 "2013-03-28T16:39:13Z")
Bastiaan Zijlema wrote: > We have an ES index with around 1 billion documents. The index > is constantly updated and expanded by 16 spiders. The spiders > send bulk inserts with curl (http api), varying between 1 and > 10,000 documents, depending on the data found by the > spiders. Everything is running fine, the inserts take a few > seconds at most. However, once in a while the curl request will > time out after 2 minutes. Which bulk requests are timing out? Is it just large ones? How many bytes are the requests that time out? -Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bastiaan\_Zijlema](https://avatars.discourse-cdn.com/v4/letter/b/977dab/32.png) [@Bastiaan\_Zijlema](https://discuss.elastic.co/u/Bastiaan_Zijlema)\ **Post date:** [March 29, 2013, 8:43am UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/3 "2013-03-29T08:43:31Z")
It really differs. Sometimes it are small ones, sometimes large ones up to 700KB in size. But when I try to reproduce the timeouts by sending large bulks (around 24MB) everything seems to be running fine. Bastiaan Op donderdag 28 maart 2013 13:54:51 UTC+1 schreef Bastiaan Zijlema het volgende: > Hello everybody, > > We have an ES index with around 1 billion documents. The index is > constantly updated and expanded by 16 spiders. The spiders send bulk > inserts with curl (http api), varying between 1 and 10,000 documents, > depending on the data found by the spiders. Everything is running fine, the > inserts take a few seconds at most. However, once in a while the curl > request will time out after 2 minutes. > > I'm trying to find out why this is happening, but I cannot find anything > useful in the logs. I only see some warnings from the garbage collector, > but not at the times the timeouts happen. So I have two questions: > > - how can I find out why the timeouts are happening (log settings)? > - Is this a bad thing, or should I just accept this will happen and > increase the curl timeout settings? > > Our setup: > > - 4 nodes > - 4 cores per node > - 24GB Ram, 16GB heap size per node > - Java 1.7 > - ES 0.20.5 > > ES settings: > > - 12 shards > - 1 replica > > Thanks in advance. > > Regards, > > Bastiaan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 29, 2013, 11:50am UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/4 "2013-03-29T11:50:40Z")
Take care your spiders don't hit the 100 MB HTTP limit which is set in http.max\_content\_length and is 100m by default. Jörg Am 29.03.13 09:43, schrieb Bastiaan Zijlema: > It really differs. Sometimes it are small ones, sometimes large ones > up to 700KB in size. But when I try to reproduce the timeouts by > sending large bulks (around 24MB) everything seems to be running fine. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bastiaan\_Zijlema](https://avatars.discourse-cdn.com/v4/letter/b/977dab/32.png) [@Bastiaan\_Zijlema](https://discuss.elastic.co/u/Bastiaan_Zijlema)\ **Post date:** [March 29, 2013, 12:23pm UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/5 "2013-03-29T12:23:10Z")
Thanks for pointing this out, I did'nt know about this. But when I try to send more than 100MB I get the error "Recv failure: Connection reset by peer when sending data", not a timeout. So this doesn't seem to be the problem causing the timeouts. Bastiaan On Friday, March 29, 2013 12:50:40 PM UTC+1, Jörg Prante wrote: > Take care your spiders don't hit the 100 MB HTTP limit which is set in > http.max\_content\_length and is 100m by default. > > Jörg > > Am 29.03.13 09:43, schrieb Bastiaan Zijlema: > > > It really differs. Sometimes it are small ones, sometimes large ones > > up to 700KB in size. But when I try to reproduce the timeouts by > > sending large bulks (around 24MB) everything seems to be running fine. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:43am UTC](https://discuss.elastic.co/t/curl-timeout-during-bulk-insert/11355/6 "2017-07-06T02:43:38Z")