# Elasticsearch-hadoop sporadic timeouts

**URL:** <https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014>\
**Category:** Elasticsearch\
**Created:** [September 30, 2014, 3:16pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014 "2014-09-30T15:16:43Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [September 30, 2014, 3:16pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/1 "2014-09-30T15:16:43Z")

</div>

Hi - we're having problems with one of our map-reduce jobs that writes to  
Elasticsearch. Lots of map tasks are failing due to ES being "unavailable",  
with logs like this:

[https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)

Seems like elasticsearch-hadoop tries talking to an ES node, it times out,  
tries the next one, it times out, etc until all nodes in the cluster are  
exhausted and then it gives up.

As far as I can tell, the ES cluster is healthy while this is occurring.  
May map tasks are succeeding - probably about 10% of the attempts are  
killed due to this issue. The main problem is that these killed tasks waste  
a lot of time, and slow down the overall job execution.

I'm not sure where to troubleshoot this next. Does anyone have any idea  
what would cause all of these time outs & failures?

I'm also curious about the lines like this:

2014-09-30 12:49:20,469 WARN org.apache.commons.httpclient.SimpleHttpConnectionManager: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.

Would that be related to the timeout problem we're seeing?

Thanks,  
Zach

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [September 30, 2014, 3:24pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/2 "2014-09-30T15:24:53Z")

</div>

What version of es-hadoop/es/cascading are you using?

On 9/30/14 6:16 PM, Zach Cox wrote:

> Hi - we're having problems with one of our map-reduce jobs that writes to Elasticsearch. Lots of map tasks are failing  
> due to ES being "unavailable", with logs like this:
> 
> [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> 
> Seems like elasticsearch-hadoop tries talking to an ES node, it times out, tries the next one, it times out, etc until  
> all nodes in the cluster are exhausted and then it gives up.
> 
> As far as I can tell, the ES cluster is healthy while this is occurring. May map tasks are succeeding - probably about  
> 10% of the attempts are killed due to this issue. The main problem is that these killed tasks waste a lot of time, and  
> slow down the overall job execution.
> 
> I'm not sure where to troubleshoot this next. Does anyone have any idea what would cause all of these time outs & failures?
> 
> I'm also curious about the lines like this:
> 
> 2014-09-30 12:49:20,469 WARN org.apache.commons.httpclient.SimpleHttpConnectionManager: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
> 
> Would that be related to the timeout problem we're seeing?
> 
> Thanks,  
> Zach
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> --  
> Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/542ACB45.3030805%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542ACB45.3030805%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [September 30, 2014, 3:43pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/3 "2014-09-30T15:43:48Z")

</div>

Hi Costin:

elasticsearch-hadoop 2.0.0  
cascading 2.5.4  
scalding 0.10.0

Thanks,  
Zach

On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:

> What version of es-hadoop/es/cascading are you using?
> 
> On 9/30/14 6:16 PM, Zach Cox wrote:
> 
> > Hi - we're having problems with one of our map-reduce jobs that writes  
> > to Elasticsearch. Lots of map tasks are failing  
> > due to ES being "unavailable", with logs like this:
> 
> [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> 
> > Seems like elasticsearch-hadoop tries talking to an ES node, it times  
> > out, tries the next one, it times out, etc until  
> > all nodes in the cluster are exhausted and then it gives up.
> > 
> > As far as I can tell, the ES cluster is healthy while this is occurring.  
> > May map tasks are succeeding - probably about  
> > 10% of the attempts are killed due to this issue. The main problem is  
> > that these killed tasks waste a lot of time, and  
> > slow down the overall job execution.
> > 
> > I'm not sure where to troubleshoot this next. Does anyone have any idea  
> > what would cause all of these time outs & failures?
> > 
> > I'm also curious about the lines like this:
> > 
> > 2014-09-30 12:49:20,469 WARN  
> > org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> > SimpleHttpConnectionManager being used incorrectly. Be sure that  
> > HttpMethod.releaseConnection() is always called and that only one thread  
> > and/or method is using this connection manager at a time.
> > 
> > Would that be related to the timeout problem we're seeing?
> > 
> > Thanks,  
> > Zach
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\> \<mailto:  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)\>.
> 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > --  
> > Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [September 30, 2014, 3:54pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/4 "2014-09-30T15:54:04Z")

</div>

Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be running into issue #256 which was fixed some time ago  
and will be part of the upcoming  
2.0.2, 2.1 Beta2.

Cheers,

On 9/30/14 6:43 PM, Zach Cox wrote:

> Hi Costin:
> 
> elasticsearch-hadoop 2.0.0  
> cascading 2.5.4  
> scalding 0.10.0
> 
> Thanks,  
> Zach
> 
> On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
> 
> ```
> What version of es-hadoop/es/cascading are you using?
> 
> On 9/30/14 6:16 PM, Zach Cox wrote:
> > Hi - we're having problems with one of our map-reduce jobs that writes to Elasticsearch. Lots of map tasks are failing
> > due to ES being "unavailable", with logs like this:
> >
> >https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
> 
> >
> > Seems like elasticsearch-hadoop tries talking to an ES node, it times out, tries the next one, it times out, etc until
> > all nodes in the cluster are exhausted and then it gives up.
> >
> > As far as I can tell, the ES cluster is healthy while this is occurring. May map tasks are succeeding - probably about
> > 10% of the attempts are killed due to this issue. The main problem is that these killed tasks waste a lot of time, and
> > slow down the overall job execution.
> >
> > I'm not sure where to troubleshoot this next. Does anyone have any idea what would cause all of these time outs & failures?
> >
> > I'm also curious about the lines like this:
> >
> > 2014-09-30 12:49:20,469 WARN org.apache.commons.httpclient.SimpleHttpConnectionManager: SimpleHttpConnectionManager being used incorrectly. Be sure that
> HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection
> manager at a time.
> >
> >
> > Would that be related to the timeout problem we're seeing?
> >
> > Thanks,
> > Zach
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> 
> > For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> --
> Costin
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/542AD21C.6030802%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542AD21C.6030802%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 1, 2014, 11:15am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/5 "2014-10-01T11:15:25Z")

</div>

Hi Costin - we updated our dependencies to use elasticsearch-hadoop  
2.0.2.BUILD-SNAPSHOT, but that didn't seem to change anything. We're still  
seeing the same task failures while trying to write to Elasticsearch. The  
only difference in the logs is that now I don't see  
the SimpleHttpConnectionManager warnings.

Any ideas what we could try next?

Thanks,  
Zach

On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:

> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be running  
> into issue #256 which was fixed some time ago  
> and will be part of the upcoming  
> 2.0.2, 2.1 Beta2.
> 
> Cheers,
> 
> On 9/30/14 6:43 PM, Zach Cox wrote:
> 
> > Hi Costin:
> > 
> > elasticsearch-hadoop 2.0.0  
> > cascading 2.5.4  
> > scalding 0.10.0
> > 
> > Thanks,  
> > Zach
> > 
> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
> > 
> > ```
> > What version of es-hadoop/es/cascading are you using? 
> > 
> > On 9/30/14 6:16 PM, Zach Cox wrote: 
> > > Hi - we're having problems with one of our map-reduce jobs that 
> > 
> > ```
> 
> writes to Elasticsearch. Lots of map tasks are failing
> 
> > ```
> > > due to ES being "unavailable", with logs like this: 
> > > 
> > >
> > 
> > ```
> 
> [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)\>
> 
> > ```
> > > 
> > > Seems like elasticsearch-hadoop tries talking to an ES node, it 
> > 
> > ```
> 
> times out, tries the next one, it times out, etc until
> 
> > ```
> > > all nodes in the cluster are exhausted and then it gives up. 
> > > 
> > > As far as I can tell, the ES cluster is healthy while this is 
> > 
> > ```
> 
> occurring. May map tasks are succeeding - probably about
> 
> > ```
> > > 10% of the attempts are killed due to this issue. The main problem 
> > 
> > ```
> 
> is that these killed tasks waste a lot of time, and
> 
> > ```
> > > slow down the overall job execution. 
> > > 
> > > I'm not sure where to troubleshoot this next. Does anyone have any 
> > 
> > ```
> 
> idea what would cause all of these time outs & failures?
> 
> > ```
> > > 
> > > I'm also curious about the lines like this: 
> > > 
> > > 2014-09-30 12:49:20,469 WARN 
> > 
> > ```
> 
> org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> SimpleHttpConnectionManager being used incorrectly. Be sure that
> 
> > ```
> > HttpMethod.releaseConnection() is always called and that only one 
> > 
> > ```
> 
> thread and/or method is using this connection
> 
> > ```
> > manager at a time. 
> > > 
> > > 
> > > Would that be related to the timeout problem we're seeing? 
> > > 
> > > Thanks, 
> > > Zach 
> > > 
> > > -- 
> > > You received this message because you are subscribed to the Google 
> > 
> > ```
> 
> Groups "elasticsearch" group.
> 
> > ```
> > > To unsubscribe from this group and stop receiving emails from it, 
> > 
> > ```
> 
> send an email to
> 
> > ```
> > >elasticsearc...@googlegroups.com <javascript:> <mailto:
> > 
> > ```
> 
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\> \<javascript:\>\>.
> 
> > ```
> > > To view this discussion on the web visit 
> > >
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)\>
> 
> > ```
> > > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)\>\>.
> 
> > ```
> > > For more options, visithttps://groups.google.com/d/optout <
> > 
> > ```
> 
> [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.
> 
> > ```
> > -- 
> > Costin 
> > 
> > ```
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\> \<mailto:  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)\>.
> 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [October 1, 2014, 12:19pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/6 "2014-10-01T12:19:55Z")

</div>

The error indicates the ES nodes don't reply in a timely fashion and thus the connection drops. Based on your logs it  
seems to be either a GC or a network issue.  
You could try turning on logging in package 'org.elasticsearch.hadoop.rest' to DEBUG.  
How many tasks do you have and what's your bulk size/entries number?

On 10/1/14 2:15 PM, Zach Cox wrote:

> Hi Costin - we updated our dependencies to use elasticsearch-hadoop 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> anything. We're still seeing the same task failures while trying to write to Elasticsearch. The only difference in the  
> logs is that now I don't see the SimpleHttpConnectionManager warnings.
> 
> Any ideas what we could try next?
> 
> Thanks,  
> Zach
> 
> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> 
> ```
> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be running into issue #256 which was fixed some time ago
> and will be part of the upcoming
> 2.0.2, 2.1 Beta2.
> 
> Cheers,
> 
> On 9/30/14 6:43 PM, Zach Cox wrote:
> > Hi Costin:
> >
> > elasticsearch-hadoop 2.0.0
> > cascading 2.5.4
> > scalding 0.10.0
> >
> > Thanks,
> > Zach
> >
> >
> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
> >
> > What version of es-hadoop/es/cascading are you using?
> >
> > On 9/30/14 6:16 PM, Zach Cox wrote:
> > > Hi - we're having problems with one of our map-reduce jobs that writes to Elasticsearch. Lots of map tasks are failing
> > > due to ES being "unavailable", with logs like this:
> > >
> > >https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
> 
> > <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>>
> 
> >
> > >
> > > Seems like elasticsearch-hadoop tries talking to an ES node, it times out, tries the next one, it times out, etc until
> > > all nodes in the cluster are exhausted and then it gives up.
> > >
> > > As far as I can tell, the ES cluster is healthy while this is occurring. May map tasks are succeeding - probably about
> > > 10% of the attempts are killed due to this issue. The main problem is that these killed tasks waste a lot of time, and
> > > slow down the overall job execution.
> > >
> > > I'm not sure where to troubleshoot this next. Does anyone have any idea what would cause all of these time outs & failures?
> > >
> > > I'm also curious about the lines like this:
> > >
> > > 2014-09-30 12:49:20,469 WARN org.apache.commons.httpclient.SimpleHttpConnectionManager: SimpleHttpConnectionManager being used incorrectly. Be sure that
> > HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection
> > manager at a time.
> > >
> > >
> > > Would that be related to the timeout problem we're seeing?
> > >
> > > Thanks,
> > > Zach
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to
> > >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>
> <javascript:>>.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>>
> > > <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>
> 
> > <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
> 
> >
> > > For more options, visithttps://groups.google.com/d/optout <http://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> 
> > For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> 
> --
> Costin
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 1, 2014, 12:41pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/7 "2014-10-01T12:41:16Z")

</div>

This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with  
total map task capacity of 25. Elasticsearch cluster has 4 nodes.

Where can I find the bulk size/entries numbers?

Thanks,  
Zach

On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:

> The error indicates the ES nodes don't reply in a timely fashion and thus  
> the connection drops. Based on your logs it seems to be either a GC or a  
> network issue.  
> You could try turning on logging in package  
> 'org.elasticsearch.hadoop.rest'  
> to DEBUG.  
> How many tasks do you have and what's your bulk size/entries number?
> 
> On 10/1/14 2:15 PM, Zach Cox wrote:
> 
> > Hi Costin - we updated our dependencies to use elasticsearch-hadoop  
> > 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> > anything. We're still seeing the same task failures while trying to write  
> > to Elasticsearch. The only difference in the  
> > logs is that now I don't see the SimpleHttpConnectionManager warnings.
> > 
> > Any ideas what we could try next?
> > 
> > Thanks,  
> > Zach
> > 
> > On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> > 
> > ```
> > Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be
> > 
> > ```
> > 
> > running into issue #256 which was fixed some time ago  
> > and will be part of the upcoming  
> > 2.0.2, 2.1 Beta2.
> > 
> > ```
> > Cheers,
> > 
> > On 9/30/14 6:43 PM, Zach Cox wrote:
> > > Hi Costin:
> > >
> > > elasticsearch-hadoop 2.0.0
> > > cascading 2.5.4
> > > scalding 0.10.0
> > >
> > > Thanks,
> > > Zach
> > >
> > >
> > > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau
> > 
> > ```

wrote:

> > ```
> > >
> > > What version of es-hadoop/es/cascading are you using?
> > >
> > > On 9/30/14 6:16 PM, Zach Cox wrote:
> > > > Hi - we're having problems with one of our map-reduce jobs
> > 
> > ```
> > 
> > that writes to Elasticsearch. Lots of map tasks are failing  
> > \> \> due to ES being "unavailable", with logs like this:  
> > \> \>  
> > \>
> > 
> > >

[https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)

> > \<  
> > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> 
> > ```
> > >
> > 
> > ```
> > 
> > \<  
> > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > 
> > \<  
> > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > 
> > ```
> > >
> > > >
> > > > Seems like elasticsearch-hadoop tries talking to an ES node,
> > 
> > ```
> > 
> > it times out, tries the next one, it times out, etc until  
> > \> \> all nodes in the cluster are exhausted and then it gives up.  
> > \> \>  
> > \> \> As far as I can tell, the ES cluster is healthy while this is  
> > occurring. May map tasks are succeeding - probably about  
> > \> \> 10% of the attempts are killed due to this issue. The main  
> > problem is that these killed tasks waste a lot of time, and  
> > \> \> slow down the overall job execution.  
> > \> \>  
> > \> \> I'm not sure where to troubleshoot this next. Does anyone  
> > have  
> > any idea what would cause all of these time outs & failures?  
> > \> \>  
> > \> \> I'm also curious about the lines like this:  
> > \> \>  
> > \> \> 2014-09-30 12:49:20,469 WARN  
> > org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> > SimpleHttpConnectionManager being used incorrectly. Be sure that  
> > \> HttpMethod.releaseConnection() is always called and that only  
> > one thread and/or method is using this connection  
> > \> manager at a time.  
> > \> \>  
> > \> \>  
> > \> \> Would that be related to the timeout problem we're seeing?  
> > \> \>  
> > \> \> Thanks,  
> > \> \> Zach  
> > \> \>  
> > \> \> --  
> > \> \> You received this message because you are subscribed to the  
> > Google Groups "elasticsearch" group.  
> > \> \> To unsubscribe from this group and stop receiving emails from  
> > it, send an email to  
> > \> \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>  
> > \<javascript:\>\>.  
> > \> \> To view this discussion on the web visit  
> > \>
> > 
> > >

[https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)

> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> 
> > ```
> > >
> > 
> > ```
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > 
> > ```
> > > >
> > 
> > ```
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> 
> > ```
> > >
> > 
> > ```
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > 
> > > .
> > 
> > ```
> > >
> > > > For more options, visithttps://groups.google.com/d/optout
> > 
> > ```
> > 
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<[https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.  
> > \> --  
> > \> Costin  
> > \>  
> > \> --  
> > \> You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > \> To unsubscribe from this group and stop receiving emails from it,  
> > send an email to  
> > \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > \> To view this discussion on the web visit
> > 
> > >

[https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)

> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> 
> > ```
> > >
> > 
> > ```
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > ```
> > > For more options, visithttps://groups.google.com/d/optout
> > 
> > ```
> > 
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > ```
> > --
> > Costin
> > 
> > ```
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)  
> > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit

[https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com)

> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe](https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit

[https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com)  
.

> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANjo42x\_aAnkbFefsbrtxx8U\_S2FAoMCY2esmsTSOPs9RfPR%2BA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42x_aAnkbFefsbrtxx8U_S2FAoMCY2esmsTSOPs9RfPR%2BA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 1, 2014, 1:00pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/8 "2014-10-01T13:00:26Z")

</div>

Hi Costin - by "bulk size/entries number" are you referring to the  
es.batch.size.bytes and es.batch.size.entries config values described here?

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

It looks like the only elasticsearch-related config values we're setting is  
this:

es.input.json = true

So we must be using default values for those es.batch.size config values.

Thanks,  
Zach

On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox [zcox522@gmail.com](mailto:zcox522@gmail.com) wrote:

> This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with  
> total map task capacity of 25. Elasticsearch cluster has 4 nodes.
> 
> Where can I find the bulk size/entries numbers?
> 
> Thanks,  
> Zach
> 
> On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:
> 
> > The error indicates the ES nodes don't reply in a timely fashion and thus  
> > the connection drops. Based on your logs it seems to be either a GC or a  
> > network issue.  
> > You could try turning on logging in package  
> > 'org.elasticsearch.hadoop.rest'  
> > to DEBUG.  
> > How many tasks do you have and what's your bulk size/entries number?
> > 
> > On 10/1/14 2:15 PM, Zach Cox wrote:
> > 
> > > Hi Costin - we updated our dependencies to use elasticsearch-hadoop  
> > > 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> > > anything. We're still seeing the same task failures while trying to  
> > > write  
> > > to Elasticsearch. The only difference in the  
> > > logs is that now I don't see the SimpleHttpConnectionManager warnings.
> > > 
> > > Any ideas what we could try next?
> > > 
> > > Thanks,  
> > > Zach
> > > 
> > > On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> > > 
> > > ```
> > > Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be
> > > 
> > > ```
> > > 
> > > running into issue #256 which was fixed some time ago  
> > > and will be part of the upcoming  
> > > 2.0.2, 2.1 Beta2.
> > > 
> > > ```
> > > Cheers,
> > > 
> > > On 9/30/14 6:43 PM, Zach Cox wrote:
> > > > Hi Costin:
> > > >
> > > > elasticsearch-hadoop 2.0.0
> > > > cascading 2.5.4
> > > > scalding 0.10.0
> > > >
> > > > Thanks,
> > > > Zach
> > > >
> > > >
> > > > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau
> > > 
> > > ```
> 
> wrote:
> 
> > > ```
> > > >
> > > > What version of es-hadoop/es/cascading are you using?
> > > >
> > > > On 9/30/14 6:16 PM, Zach Cox wrote:
> > > > > Hi - we're having problems with one of our map-reduce jobs
> > > 
> > > ```
> > > 
> > > that writes to Elasticsearch. Lots of map tasks are failing  
> > > \> \> due to ES being "unavailable", with logs like this:  
> > > \> \>  
> > > \>
> > > 
> > > >
> 
> [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> 
> > > \<  
> > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > 
> > > ```
> > > >
> > > 
> > > ```
> > > 
> > > \<  
> > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > > 
> > > \<  
> > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > > 
> > > ```
> > > >
> > > > >
> > > > > Seems like elasticsearch-hadoop tries talking to an ES node,
> > > 
> > > ```
> > > 
> > > it times out, tries the next one, it times out, etc until  
> > > \> \> all nodes in the cluster are exhausted and then it gives up.  
> > > \> \>  
> > > \> \> As far as I can tell, the ES cluster is healthy while this  
> > > is  
> > > occurring. May map tasks are succeeding - probably about  
> > > \> \> 10% of the attempts are killed due to this issue. The main  
> > > problem is that these killed tasks waste a lot of time, and  
> > > \> \> slow down the overall job execution.  
> > > \> \>  
> > > \> \> I'm not sure where to troubleshoot this next. Does anyone  
> > > have  
> > > any idea what would cause all of these time outs & failures?  
> > > \> \>  
> > > \> \> I'm also curious about the lines like this:  
> > > \> \>  
> > > \> \> 2014-09-30 12:49:20,469 WARN  
> > > org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> > > SimpleHttpConnectionManager being used incorrectly. Be sure that  
> > > \> HttpMethod.releaseConnection() is always called and that only  
> > > one thread and/or method is using this connection  
> > > \> manager at a time.  
> > > \> \>  
> > > \> \>  
> > > \> \> Would that be related to the timeout problem we're seeing?  
> > > \> \>  
> > > \> \> Thanks,  
> > > \> \> Zach  
> > > \> \>  
> > > \> \> --  
> > > \> \> You received this message because you are subscribed to the  
> > > Google Groups "elasticsearch" group.  
> > > \> \> To unsubscribe from this group and stop receiving emails  
> > > from  
> > > it, send an email to  
> > > \> \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>  
> > > \<javascript:\>\>.  
> > > \> \> To view this discussion on the web visit  
> > > \>
> > > 
> > > >
> 
> [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > 
> > > ```
> > > >
> > > 
> > > ```
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > > 
> > > ```
> > > > >
> > > 
> > > ```
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > 
> > > ```
> > > >
> > > 
> > > ```
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > 
> > > > .
> > > 
> > > ```
> > > >
> > > > > For more options, visithttps://groups.google.com/d/optout
> > > 
> > > ```
> > > 
> > > [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<[https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.  
> > > \> --  
> > > \> Costin  
> > > \>  
> > > \> --  
> > > \> You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > \> To unsubscribe from this group and stop receiving emails from it,  
> > > send an email to  
> > > \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > > \> To view this discussion on the web visit
> > > 
> > > >
> 
> [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> > 
> > > ```
> > > >
> > > 
> > > ```
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > ```
> > > > For more options, visithttps://groups.google.com/d/optout
> > > 
> > > ```
> > > 
> > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > ```
> > > --
> > > Costin
> > > 
> > > ```
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an  
> > > email to  
> > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)  
> > > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com)
> 
> > > \<  
> > > [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > Costin
> > 
> > --  
> > You received this message because you are subscribed to a topic in the  
> > Google Groups "elasticsearch" group.  
> > To unsubscribe from this topic, visit  
> > [https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe](https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe).  
> > To unsubscribe from this group and all its topics, send an email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com)  
> .
> 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANjo42wT\_Govm3zL%2ByhGvO--wA1%2BJS2FPSgbxj5bfdcM5Aghgg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42wT_Govm3zL%2ByhGvO--wA1%2BJS2FPSgbxj5bfdcM5Aghgg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 3, 2014, 3:09pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/9 "2014-10-03T15:09:17Z")

</div>

Is there anything else we could try here to debug elasticsearch-hadoop  
being unable to write to Elasticsearch? We're still seeing the same number  
of these fails during the nightly batch runs even after switching to  
2.0.2.BUILD-SNAPSHOT,  
and I don't see any additional lines from org.elasticsearch.hadoop.rest in  
the tasktracker logs after adding to our logback.xml.

Here are example logs for a task that failed this morning:  
[https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt](https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt)

Is it reasonable to expect 4 Elasticsearch nodes to handle the batch write  
volume from 5 Hadoop nodes (25 map tasks)?

Thanks,  
Zach

On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox [zcox522@gmail.com](mailto:zcox522@gmail.com) wrote:

> Hi Costin - by "bulk size/entries number" are you referring to the  
> es.batch.size.bytes and es.batch.size.entries config values described here?
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#configuration-serialization)
> 
> It looks like the only elasticsearch-related config values we're setting  
> is this:
> 
> es.input.json = true
> 
> So we must be using default values for those es.batch.size config values.
> 
> Thanks,  
> Zach
> 
> On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox [zcox522@gmail.com](mailto:zcox522@gmail.com) wrote:
> 
> > This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with  
> > total map task capacity of 25. Elasticsearch cluster has 4 nodes.
> > 
> > Where can I find the bulk size/entries numbers?
> > 
> > Thanks,  
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com)  
> > wrote:
> > 
> > > The error indicates the ES nodes don't reply in a timely fashion and  
> > > thus  
> > > the connection drops. Based on your logs it seems to be either a GC or a  
> > > network issue.  
> > > You could try turning on logging in package  
> > > 'org.elasticsearch.hadoop.rest'  
> > > to DEBUG.  
> > > How many tasks do you have and what's your bulk size/entries number?
> > > 
> > > On 10/1/14 2:15 PM, Zach Cox wrote:
> > > 
> > > > Hi Costin - we updated our dependencies to use elasticsearch-hadoop  
> > > > 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> > > > anything. We're still seeing the same task failures while trying to  
> > > > write  
> > > > to Elasticsearch. The only difference in the  
> > > > logs is that now I don't see the SimpleHttpConnectionManager warnings.
> > > > 
> > > > Any ideas what we could try next?
> > > > 
> > > > Thanks,  
> > > > Zach
> > > > 
> > > > On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> > > > 
> > > > ```
> > > > Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be
> > > > 
> > > > ```
> > > > 
> > > > running into issue #256 which was fixed some time ago  
> > > > and will be part of the upcoming  
> > > > 2.0.2, 2.1 Beta2.
> > > > 
> > > > ```
> > > > Cheers,
> > > > 
> > > > On 9/30/14 6:43 PM, Zach Cox wrote:
> > > > > Hi Costin:
> > > > >
> > > > > elasticsearch-hadoop 2.0.0
> > > > > cascading 2.5.4
> > > > > scalding 0.10.0
> > > > >
> > > > > Thanks,
> > > > > Zach
> > > > >
> > > > >
> > > > > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau
> > > > 
> > > > ```
> > 
> > wrote:
> > 
> > > > ```
> > > > >
> > > > > What version of es-hadoop/es/cascading are you using?
> > > > >
> > > > > On 9/30/14 6:16 PM, Zach Cox wrote:
> > > > > > Hi - we're having problems with one of our map-reduce jobs
> > > > 
> > > > ```
> > > > 
> > > > that writes to Elasticsearch. Lots of map tasks are failing  
> > > > \> \> due to ES being "unavailable", with logs like this:  
> > > > \> \>  
> > > > \>
> > > > 
> > > > >
> > 
> > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > 
> > > > \<  
> > > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > > 
> > > > ```
> > > > >
> > > > 
> > > > ```
> > > > 
> > > > \<  
> > > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > > > 
> > > > \<  
> > > > [https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt](https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt)
> > > > 
> > > > ```
> > > > >
> > > > > >
> > > > > > Seems like elasticsearch-hadoop tries talking to an ES
> > > > 
> > > > ```
> > 
> > node,
> > 
> > > > it times out, tries the next one, it times out, etc until  
> > > > \> \> all nodes in the cluster are exhausted and then it gives  
> > > > up.  
> > > > \> \>  
> > > > \> \> As far as I can tell, the ES cluster is healthy while this  
> > > > is  
> > > > occurring. May map tasks are succeeding - probably about  
> > > > \> \> 10% of the attempts are killed due to this issue. The main  
> > > > problem is that these killed tasks waste a lot of time, and  
> > > > \> \> slow down the overall job execution.  
> > > > \> \>  
> > > > \> \> I'm not sure where to troubleshoot this next. Does anyone  
> > > > have  
> > > > any idea what would cause all of these time outs & failures?  
> > > > \> \>  
> > > > \> \> I'm also curious about the lines like this:  
> > > > \> \>  
> > > > \> \> 2014-09-30 12:49:20,469 WARN  
> > > > org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> > > > SimpleHttpConnectionManager being used incorrectly. Be sure that  
> > > > \> HttpMethod.releaseConnection() is always called and that only  
> > > > one thread and/or method is using this connection  
> > > > \> manager at a time.  
> > > > \> \>  
> > > > \> \>  
> > > > \> \> Would that be related to the timeout problem we're seeing?  
> > > > \> \>  
> > > > \> \> Thanks,  
> > > > \> \> Zach  
> > > > \> \>  
> > > > \> \> --  
> > > > \> \> You received this message because you are subscribed to the  
> > > > Google Groups "elasticsearch" group.  
> > > > \> \> To unsubscribe from this group and stop receiving emails  
> > > > from  
> > > > it, send an email to  
> > > > \> \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > > > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>  
> > > > \<javascript:\>\>.  
> > > > \> \> To view this discussion on the web visit  
> > > > \>
> > > > 
> > > > >
> > 
> > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > > 
> > > > ```
> > > > >
> > > > 
> > > > ```
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com)
> > > > 
> > > > ```
> > > > > >
> > > > 
> > > > ```
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > 
> > > > ```
> > > > >
> > > > 
> > > > ```
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > 
> > > > > .
> > > > 
> > > > ```
> > > > >
> > > > > > For more options, visithttps://groups.google.com/d/optout
> > > > 
> > > > ```
> > > > 
> > > > [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<  
> > > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.  
> > > > \> --  
> > > > \> Costin  
> > > > \>  
> > > > \> --  
> > > > \> You received this message because you are subscribed to the  
> > > > Google  
> > > > Groups "elasticsearch" group.  
> > > > \> To unsubscribe from this group and stop receiving emails from it,  
> > > > send an email to  
> > > > \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>  
> > > > \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > > > \> To view this discussion on the web visit
> > > > 
> > > > >
> > 
> > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com)
> > > 
> > > > ```
> > > > >
> > > > 
> > > > ```
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > ```
> > > > > For more options, visithttps://groups.google.com/d/optout
> > > > 
> > > > ```
> > > > 
> > > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > ```
> > > > --
> > > > Costin
> > > > 
> > > > ```
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups  
> > > > "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an  
> > > > email to  
> > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)  
> > > > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > > To view this discussion on the web visit
> > 
> > [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com)
> > 
> > > > \<  
> > > > [https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > Costin
> > > 
> > > --  
> > > You received this message because you are subscribed to a topic in the  
> > > Google Groups "elasticsearch" group.  
> > > To unsubscribe from this topic, visit  
> > > [https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe](https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe)  
> > > .  
> > > To unsubscribe from this group and all its topics, send an email to  
> > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit
> > 
> > [https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com)  
> > .
> > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [October 3, 2014, 3:46pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/10 "2014-10-03T15:46:02Z")

</div>

You can always enable TRACE though that is likely to create way too much information in production and slow things down  
considerably.

The first thing you can do is minimize the batch size to give ES more breathing space by minimizing the batch size (say  
to 512KB) or the number  
of entries (500 instead of 1k, etc...).  
However the error indicates a network issue not an Elasticsearch one so check whether there's some type of  
service/network/firewall initialization happening at night. Do the errors occur around the same time? Is there some  
backup procedure that potentially kicks in?

Potentially you can try and increase the default http timeout (es.http.timeout) from 1m to 3m or so. However this is  
really a patch since if the ES server doesn't return a response in 1m, it means things are not going well at all.

On 10/3/14 6:09 PM, Zach Cox wrote:

> Is there anything else we could try here to debug elasticsearch-hadoop being unable to write to Elasticsearch? We're  
> still seeing the same number of these fails during the nightly batch runs even after switching to 2.0.2.BUILD-SNAPSHOT,  
> and I don't see any additional lines from org.elasticsearch.hadoop.\_\_rest in the tasktracker logs after adding to our logback.xml.
> 
> Here are example logs for a task that failed this morning:  
> [https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt](https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt)
> 
> Is it reasonable to expect 4 Elasticsearch nodes to handle the batch write volume from 5 Hadoop nodes (25 map tasks)?
> 
> Thanks,  
> Zach
> 
> On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox \<[zcox522@gmail.com](mailto:zcox522@gmail.com) [mailto:zcox522@gmail.com](mailto:zcox522@gmail.com)\> wrote:
> 
> ```
> Hi Costin - by "bulk size/entries number" are you referring to the es.batch.size.bytes and es.batch.size.entries
> config values described here?
> 
> http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#configuration-serialization
> 
> It looks like the only elasticsearch-related config values we're setting is this:
> 
> es.input.json = true
> 
> So we must be using default values for those es.batch.size config values.
> 
> Thanks,
> Zach
> 
> On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <zcox522@gmail.com <mailto:zcox522@gmail.com>> wrote:
> 
> This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with total map task capacity of 25.
> Elasticsearch cluster has 4 nodes.
> 
> Where can I find the bulk size/entries numbers?
> 
> Thanks,
> Zach
> 
> On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <costin.leau@gmail.com <mailto:costin.leau@gmail.com>> wrote:
> > The error indicates the ES nodes don't reply in a timely fashion and thus
> > the connection drops. Based on your logs it seems to be either a GC or a
> > network issue.
> > You could try turning on logging in package 'org.elasticsearch.hadoop.rest'
> > to DEBUG.
> > How many tasks do you have and what's your bulk size/entries number?
> >
> >
> > On 10/1/14 2:15 PM, Zach Cox wrote:
> >>
> >> Hi Costin - we updated our dependencies to use elasticsearch-hadoop
> >> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change
> >> anything. We're still seeing the same task failures while trying to write
> >> to Elasticsearch. The only difference in the
> >> logs is that now I don't see the SimpleHttpConnectionManager warnings.
> >>
> >> Any ideas what we could try next?
> >>
> >> Thanks,
> >> Zach
> >>
> >>
> >> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> >>
> >> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be
> >> running into issue #256 which was fixed some time ago
> >> and will be part of the upcoming
> >> 2.0.2, 2.1 Beta2.
> >>
> >> Cheers,
> >>
> >> On 9/30/14 6:43 PM, Zach Cox wrote:
> >> > Hi Costin:
> >> >
> >> > elasticsearch-hadoop 2.0.0
> >> > cascading 2.5.4
> >> > scalding 0.10.0
> >> >
> >> > Thanks,
> >> > Zach
> >> >
> >> >
> >> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
> >> >
> >> > What version of es-hadoop/es/cascading are you using?
> >> >
> >> > On 9/30/14 6:16 PM, Zach Cox wrote:
> >> > > Hi - we're having problems with one of our map-reduce jobs
> >> that writes to Elasticsearch. Lots of map tasks are failing
> >> > > due to ES being "unavailable", with logs like this:
> >> > >
> >> >
> >> >https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
> >>
> >> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
> >>
> >> >
> >> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
> >>
> >> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>>
> >>
> >> >
> >> > >
> >> > > Seems like elasticsearch-hadoop tries talking to an ES node,
> >> it times out, tries the next one, it times out, etc until
> >> > > all nodes in the cluster are exhausted and then it gives up.
> >> > >
> >> > > As far as I can tell, the ES cluster is healthy while this is
> >> occurring. May map tasks are succeeding - probably about
> >> > > 10% of the attempts are killed due to this issue. The main
> >> problem is that these killed tasks waste a lot of time, and
> >> > > slow down the overall job execution.
> >> > >
> >> > > I'm not sure where to troubleshoot this next. Does anyone have
> >> any idea what would cause all of these time outs & failures?
> >> > >
> >> > > I'm also curious about the lines like this:
> >> > >
> >> > > 2014-09-30 12:49:20,469 WARN
> >> org.apache.commons.httpclient.SimpleHttpConnectionManager:
> >> SimpleHttpConnectionManager being used incorrectly. Be sure that
> >> > HttpMethod.releaseConnection() is always called and that only
> >> one thread and/or method is using this connection
> >> > manager at a time.
> >> > >
> >> > >
> >> > > Would that be related to the timeout problem we're seeing?
> >> > >
> >> > > Thanks,
> >> > > Zach
> >> > >
> >> > > --
> >> > > You received this message because you are subscribed to the
> >> Google Groups "elasticsearch" group.
> >> > > To unsubscribe from this group and stop receiving emails from
> >> it, send an email to
> >> > >elasticsearc...@googlegroups.com <mailto:elasticsearc...@googlegroups.com> <javascript:>
> >> <mailto:elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <javascript:>
> >> <javascript:>>.
> >> > > To view this discussion on the web visit
> >> >
> >> >https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
> >> >
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>>
> >> > >
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>
> >> >
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
> >>
> >> >
> >> > > For more options, visithttps://groups.google.com/d/optout <http://groups.google.com/d/optout>
> >> <http://groups.google.com/d/optout> <https://groups.google.com/d/optout
> >> <https://groups.google.com/d/optout>>.
> >> > --
> >> > Costin
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> Groups "elasticsearch" group.
> >> > To unsubscribe from this group and stop receiving emails from it,
> >> send an email to
> >> >elasticsearc...@googlegroups.com <mailto:elasticsearc...@googlegroups.com> <javascript:>
> >> <mailto:elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <javascript:>>.
> >> > To view this discussion on the web visit
> >>
> >> >https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com>
> >> >
> >> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >>
> >> > For more options, visithttps://groups.google.com/d/optout <http://groups.google.com/d/optout>
> >> <https://groups.google.com/d/optout>.
> >>
> >> --
> >> Costin
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "elasticsearch" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to
> >>elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>
> >> <mailto:elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>>.
> >> To view this discussion on the web visit
> >>
> >>https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com
> >>
> >> <https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >> For more options, visithttps://groups.google.com/d/optout.
> >
> >
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> >https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to
> >elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com.
> >
> > For more options, visithttps://groups.google.com/d/optout.
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/542EC4BA.60204%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542EC4BA.60204%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 3, 2014, 3:59pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/11 "2014-10-03T15:59:54Z")

</div>

Our Hadoop and Elasticsearch are all on AWS. We have 2 MR jobs that write  
to ES - 1 of them works fine, and one of them takes forever due to 10-20%  
of tasks failing in the way I've described. So I don't think it's any kind  
of network/firewall issue. There are no nightly backups related to ES or  
anything.

Would 25 map tasks batch writing to a 4-node Elasticsearch cluster cause it  
so much pain that it stops responding to writes? If I ssh to the ES nodes  
while this job is running and some tasks are failing, it doesn't seem like  
ES is under much stress.

Thanks,  
Zach

On Fri, Oct 3, 2014 at 10:46 AM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:

> You can always enable TRACE though that is likely to create way too much  
> information in production and slow things down considerably.
> 
> The first thing you can do is minimize the batch size to give ES more  
> breathing space by minimizing the batch size (say to 512KB) or the number  
> of entries (500 instead of 1k, etc...).  
> However the error indicates a network issue not an Elasticsearch one so  
> check whether there's some type of service/network/firewall initialization  
> happening at night. Do the errors occur around the same time? Is there some  
> backup procedure that potentially kicks in?
> 
> Potentially you can try and increase the default http timeout  
> (es.http.timeout) from 1m to 3m or so. However this is really a patch since  
> if the ES server doesn't return a response in 1m, it means things are not  
> going well at all.
> 
> On 10/3/14 6:09 PM, Zach Cox wrote:
> 
> > Is there anything else we could try here to debug elasticsearch-hadoop  
> > being unable to write to Elasticsearch? We're  
> > still seeing the same number of these fails during the nightly batch runs  
> > even after switching to 2.0.2.BUILD-SNAPSHOT,  
> > and I don't see any additional lines from org.elasticsearch.hadoop.\_\_rest  
> > in the tasktracker logs after adding to our logback.xml.
> > 
> > Here are example logs for a task that failed this morning:  
> > [https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/](https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/)  
> > ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt
> > 
> > Is it reasonable to expect 4 Elasticsearch nodes to handle the batch  
> > write volume from 5 Hadoop nodes (25 map tasks)?
> > 
> > Thanks,  
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox \<[zcox522@gmail.com](mailto:zcox522@gmail.com) \<mailto:  
> > [zcox522@gmail.com](mailto:zcox522@gmail.com)\>\> wrote:
> > 
> > ```
> > Hi Costin - by "bulk size/entries number" are you referring to the
> > 
> > ```
> > 
> > es.batch.size.bytes and es.batch.size.entries  
> > config values described here?
> > 
> > ```
> > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/
> > 
> > ```
> > 
> > master/configuration.html#configuration-serialization
> > 
> > ```
> > It looks like the only elasticsearch-related config values we're
> > 
> > ```
> > 
> > setting is this:
> > 
> > ```
> > es.input.json = true
> > 
> > So we must be using default values for those es.batch.size config
> > 
> > ```
> > 
> > values.
> > 
> > ```
> > Thanks,
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <zcox522@gmail.com <mailto:
> > 
> > ```
> > 
> > [zcox522@gmail.com](mailto:zcox522@gmail.com)\>\> wrote:
> > 
> > ```
> > This particular job has 1353 map tasks, Hadoop cluster has 5
> > 
> > ```
> > 
> > nodes with total map task capacity of 25.  
> > Elasticsearch cluster has 4 nodes.
> > 
> > ```
> > Where can I find the bulk size/entries numbers?
> > 
> > Thanks,
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <
> > 
> > ```
> > 
> > [costin.leau@gmail.com](mailto:costin.leau@gmail.com) [mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com)\> wrote:  
> > \> The error indicates the ES nodes don't reply in a timely  
> > fashion and thus  
> > \> the connection drops. Based on your logs it seems to be either  
> > a GC or a  
> > \> network issue.  
> > \> You could try turning on logging in package  
> > 'org.elasticsearch.hadoop.rest'  
> > \> to DEBUG.  
> > \> How many tasks do you have and what's your bulk size/entries  
> > number?  
> > \>  
> > \>  
> > \> On 10/1/14 2:15 PM, Zach Cox wrote:  
> > \>\>  
> > \>\> Hi Costin - we updated our dependencies to use  
> > elasticsearch-hadoop  
> > \>\> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> > \>\> anything. We're still seeing the same task failures while  
> > trying to write  
> > \>\> to Elasticsearch. The only difference in the  
> > \>\> logs is that now I don't see the SimpleHttpConnectionManager  
> > warnings.  
> > \>\>  
> > \>\> Any ideas what we could try next?  
> > \>\>  
> > \>\> Thanks,  
> > \>\> Zach  
> > \>\>  
> > \>\>  
> > \>\> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau  
> > wrote:  
> > \>\>  
> > \>\> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you  
> > might be  
> > \>\> running into issue #256 which was fixed some time ago  
> > \>\> and will be part of the upcoming  
> > \>\> 2.0.2, 2.1 Beta2.  
> > \>\>  
> > \>\> Cheers,  
> > \>\>  
> > \>\> On 9/30/14 6:43 PM, Zach Cox wrote:  
> > \>\> \> Hi Costin:  
> > \>\> \>  
> > \>\> \> elasticsearch-hadoop 2.0.0  
> > \>\> \> cascading 2.5.4  
> > \>\> \> scalding 0.10.0  
> > \>\> \>  
> > \>\> \> Thanks,  
> > \>\> \> Zach  
> > \>\> \>  
> > \>\> \>  
> > \>\> \> On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin  
> > Leau wrote:  
> > \>\> \>  
> > \>\> \> What version of es-hadoop/es/cascading are you using?  
> > \>\> \>  
> > \>\> \> On 9/30/14 6:16 PM, Zach Cox wrote:  
> > \>\> \> \> Hi - we're having problems with one of our  
> > map-reduce jobs  
> > \>\> that writes to Elasticsearch. Lots of map tasks are failing  
> > \>\> \> \> due to ES being "unavailable", with logs like this:  
> > \>\> \> \>  
> > \>\> \>  
> > \>\> \>[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt  
> > \>\>  
> > \>\> \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>  
> > \>\>  
> > \>\> \>  
> > \>\> \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt  
> > \>\>  
> > \>\> \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>\>  
> > \>\>  
> > \>\> \>  
> > \>\> \> \>  
> > \>\> \> \> Seems like elasticsearch-hadoop tries talking to  
> > an ES node,  
> > \>\> it times out, tries the next one, it times out, etc until  
> > \>\> \> \> all nodes in the cluster are exhausted and then it  
> > gives up.  
> > \>\> \> \>  
> > \>\> \> \> As far as I can tell, the ES cluster is healthy  
> > while this is  
> > \>\> occurring. May map tasks are succeeding - probably about  
> > \>\> \> \> 10% of the attempts are killed due to this issue.  
> > The main  
> > \>\> problem is that these killed tasks waste a lot of time, and  
> > \>\> \> \> slow down the overall job execution.  
> > \>\> \> \>  
> > \>\> \> \> I'm not sure where to troubleshoot this next. Does  
> > anyone have  
> > \>\> any idea what would cause all of these time outs & failures?  
> > \>\> \> \>  
> > \>\> \> \> I'm also curious about the lines like this:  
> > \>\> \> \>  
> > \>\> \> \> 2014-09-30 12:49:20,469 WARN  
> > \>\> org.apache.commons.httpclient.SimpleHttpConnectionManager:  
> > \>\> SimpleHttpConnectionManager being used incorrectly. Be sure  
> > that  
> > \>\> \> HttpMethod.releaseConnection() is always called and  
> > that only  
> > \>\> one thread and/or method is using this connection  
> > \>\> \> manager at a time.  
> > \>\> \> \>  
> > \>\> \> \>  
> > \>\> \> \> Would that be related to the timeout problem we're  
> > seeing?  
> > \>\> \> \>  
> > \>\> \> \> Thanks,  
> > \>\> \> \> Zach  
> > \>\> \> \>  
> > \>\> \> \> --  
> > \>\> \> \> You received this message because you are  
> > subscribed to the  
> > \>\> Google Groups "elasticsearch" group.  
> > \>\> \> \> To unsubscribe from this group and stop receiving  
> > emails from  
> > \>\> it, send an email to  
> > \>\> \> \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<mailto:  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\> \<javascript:\>  
> > \>\> \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\> \<javascript:\>  
> > \>\> \<javascript:\>\>.  
> > \>\> \> \> To view this discussion on the web visit  
> > \>\> \>  
> > \>\> \>[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\> \>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>\>  
> > \>\> \> \>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>  
> > \>\>  
> > \>\> \>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>\>.  
> > \>\>  
> > \>\> \>  
> > \>\> \> \> For more options, visithttps://groups.google.  
> > com/d/optout [http://groups.google.com/d/optout](http://groups.google.com/d/optout)  
> > \>\> [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > \>\> [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.  
> > \>\> \> --  
> > \>\> \> Costin  
> > \>\> \>  
> > \>\> \> --  
> > \>\> \> You received this message because you are subscribed to  
> > the Google  
> > \>\> Groups "elasticsearch" group.  
> > \>\> \> To unsubscribe from this group and stop receiving emails  
> > from it,  
> > \>\> send an email to  
> > \>\> \>[elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<mailto:elasticsearc...@  
> > [googlegroups.com](http://googlegroups.com)\> \<javascript:\>  
> > \>\> \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\> \<javascript:\>\>.  
> > \>\> \> To view this discussion on the web visit  
> > \>\>  
> > \>\> \>[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40googlegroups.com](http://40googlegroups.com)  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\> \>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>.  
> > \>\>  
> > \>\> \> For more options, visithttps://groups.google.  
> > com/d/optout [http://groups.google.com/d/optout](http://groups.google.com/d/optout)  
> > \>\> [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > \>\>  
> > \>\> --  
> > \>\> Costin  
> > \>\>  
> > \>\> --  
> > \>\> You received this message because you are subscribed to the  
> > Google Groups  
> > \>\> "elasticsearch" group.  
> > \>\> To unsubscribe from this group and stop receiving emails from  
> > it, send an  
> > \>\> email to  
> > \>\>[elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>  
> > \>\> \<[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>\>.  
> > \>\> To view this discussion on the web visit  
> > \>\>  
> > \>\>[https://groups.google.com/d/msgid/elasticsearch/90b059ef-](https://groups.google.com/d/msgid/elasticsearch/90b059ef-)  
> > 680e-4b5c-a0c9-dc5e5038205a%[40googlegroups.com](http://40googlegroups.com)  
> > \>\>  
> > \>\> \<[https://groups.google.com/d/msgid/elasticsearch/90b059ef-](https://groups.google.com/d/msgid/elasticsearch/90b059ef-)  
> > 680e-4b5c-a0c9-dc5e5038205a%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>.  
> > \>\> For more options, visithttps://groups.google.com/d/optout.  
> > \>  
> > \>  
> > \> --  
> > \> Costin  
> > \>  
> > \> --  
> > \> You received this message because you are subscribed to a topic  
> > in the  
> > \> Google Groups "elasticsearch" group.  
> > \> To unsubscribe from this topic, visit  
> > \>[https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > BKR18lczF1w/unsubscribe.  
> > \> To unsubscribe from this group and all its topics, send an  
> > email to  
> > \>[elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>.  
> > \> To view this discussion on the web visit  
> > \>[https://groups.google.com/d/msgid/elasticsearch/542BF16B](https://groups.google.com/d/msgid/elasticsearch/542BF16B).  
> > 6010103%[40gmail.com](http://40gmail.com).  
> > \>  
> > \> For more options, visithttps://groups.google.com/d/optout.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:elasticsearch+  
> > [unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com)\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%](https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%25)  
> > 3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%[40mail.gmail.com](http://40mail.gmail.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-  
> > A8w%[40mail.gmail.com?utm\_medium=email&utm\_source=footer](http://40mail.gmail.com?utm_medium=email&utm_source=footer)\>.  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> topic/elasticsearch/BKR18lczF1w/unsubscribe.  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> msgid/elasticsearch/542EC4BA.60204%[40gmail.com](http://40gmail.com).
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [October 3, 2014, 4:27pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/12 "2014-10-03T16:27:50Z")

</div>

What type of AWS instances are you using? Virtualization tends to interfere in various ways with a running system -  
sometime for good, sometimes for worse.

The number of tasks is good to compute the total number of data and entries you are throwing at ES at one time. You are  
looking at a maximum of 25 MBs or 25K entries against the 4 node Elastic. Depending on how beefy the machines are and  
what type of storage and mapping you have, it might be quick or slow - we can only guess. Try using Marvel or other  
monitoring tools so you can see whether there's load building up or if anything unusual happens.

Since it's unclear what the issue might be, take baby steps [1] and start with minimal load (smaller bulk size + less  
tasks) see whether there are any issues and keep on going.

[1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/troubleshooting.html)

On 10/3/14 6:59 PM, Zach Cox wrote:

> Our Hadoop and Elasticsearch are all on AWS. We have 2 MR jobs that write to ES - 1 of them works fine, and one of them  
> takes forever due to 10-20% of tasks failing in the way I've described. So I don't think it's any kind of  
> network/firewall issue. There are no nightly backups related to ES or anything.
> 
> Would 25 map tasks batch writing to a 4-node Elasticsearch cluster cause it so much pain that it stops responding to  
> writes? If I ssh to the ES nodes while this job is running and some tasks are failing, it doesn't seem like ES is under  
> much stress.
> 
> Thanks,  
> Zach
> 
> On Fri, Oct 3, 2014 at 10:46 AM, Costin Leau \<[costin.leau@gmail.com](mailto:costin.leau@gmail.com) [mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com)\> wrote:
> 
> ```
> You can always enable TRACE though that is likely to create way too much information in production and slow things
> down considerably.
> 
> The first thing you can do is minimize the batch size to give ES more breathing space by minimizing the batch size
> (say to 512KB) or the number
> of entries (500 instead of 1k, etc...).
> However the error indicates a network issue not an Elasticsearch one so check whether there's some type of
> service/network/firewall initialization happening at night. Do the errors occur around the same time? Is there some
> backup procedure that potentially kicks in?
> 
> Potentially you can try and increase the default http timeout (es.http.timeout) from 1m to 3m or so. However this is
> really a patch since if the ES server doesn't return a response in 1m, it means things are not going well at all.
> 
> On 10/3/14 6:09 PM, Zach Cox wrote:
> 
> Is there anything else we could try here to debug elasticsearch-hadoop being unable to write to Elasticsearch? We're
> still seeing the same number of these fails during the nightly batch runs even after switching to
> 2.0.2.BUILD-SNAPSHOT,
> and I don't see any additional lines from org.elasticsearch.hadoop. ____ rest in the tasktracker logs after adding
> <logger
> name="org.elasticsearch.__hadoop.rest" level="DEBUG"/> to our logback.xml.
> 
> Here are example logs for a task that failed this morning:
> https://gist. __githubusercontent.com/zcox/__ c5c81f4f8ee26d7bedbf/raw/ __ac9bef7bfe3a8e1ef69a62aabe4c7c__ 3983882f19/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt>
> 
> Is it reasonable to expect 4 Elasticsearch nodes to handle the batch write volume from 5 Hadoop nodes (25 map
> tasks)?
> 
> Thanks,
> Zach
> 
> On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox <zcox522@gmail.com <mailto:zcox522@gmail.com> <mailto:zcox522@gmail.com
> <mailto:zcox522@gmail.com>>> wrote:
> 
> Hi Costin - by "bulk size/entries number" are you referring to the es.batch.size.bytes and
> es.batch.size.entries
> config values described here?
> 
> http://www.elasticsearch.org/ __guide/en/elasticsearch/hadoop/__ master/configuration.html#__configuration-serialization
> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#configuration-serialization>
> 
> It looks like the only elasticsearch-related config values we're setting is this:
> 
> es.input.json = true
> 
> So we must be using default values for those es.batch.size config values.
> 
> Thanks,
> Zach
> 
> On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <zcox522@gmail.com <mailto:zcox522@gmail.com>
> <mailto:zcox522@gmail.com <mailto:zcox522@gmail.com>>> wrote:
> 
> This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with total map task capacity of 25.
> Elasticsearch cluster has 4 nodes.
> 
> Where can I find the bulk size/entries numbers?
> 
> Thanks,
> Zach
> 
> On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <costin.leau@gmail.com <mailto:costin.leau@gmail.com>
> <mailto:costin.leau@gmail.com <mailto:costin.leau@gmail.com>>__> wrote:
> > The error indicates the ES nodes don't reply in a timely fashion and thus
> > the connection drops. Based on your logs it seems to be either a GC or a
> > network issue.
> > You could try turning on logging in package 'org.elasticsearch.hadoop.__rest'
> > to DEBUG.
> > How many tasks do you have and what's your bulk size/entries number?
> >
> >
> > On 10/1/14 2:15 PM, Zach Cox wrote:
> >>
> >> Hi Costin - we updated our dependencies to use elasticsearch-hadoop
> >> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change
> >> anything. We're still seeing the same task failures while trying to write
> >> to Elasticsearch. The only difference in the
> >> logs is that now I don't see the SimpleHttpConnectionManager warnings.
> >>
> >> Any ideas what we could try next?
> >>
> >> Thanks,
> >> Zach
> >>
> >>
> >> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
> >>
> >> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be
> >> running into issue #256 which was fixed some time ago
> >> and will be part of the upcoming
> >> 2.0.2, 2.1 Beta2.
> >>
> >> Cheers,
> >>
> >> On 9/30/14 6:43 PM, Zach Cox wrote:
> >> > Hi Costin:
> >> >
> >> > elasticsearch-hadoop 2.0.0
> >> > cascading 2.5.4
> >> > scalding 0.10.0
> >> >
> >> > Thanks,
> >> > Zach
> >> >
> >> >
> >> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
> >> >
> >> > What version of es-hadoop/es/cascading are you using?
> >> >
> >> > On 9/30/14 6:16 PM, Zach Cox wrote:
> >> > > Hi - we're having problems with one of our map-reduce jobs
> >> that writes to Elasticsearch. Lots of map tasks are failing
> >> > > due to ES being "unavailable", with logs like this:
> >> > >
> >> >
> >>
> >https://gist. __githubusercontent.com/zcox/__ 3d6cf4329d49ca03271b/raw/ __57c46a5e4c9ea04d5c4209414d6f84__ 7492d16c0d/gistfile1.txt <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
> >>
> >>
> <https://gist. __githubusercontent.com/zcox/__ 3d6cf4329d49ca03271b/raw/ __57c46a5e4c9ea04d5c4209414d6f84__ 7492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>>
> >>
> >> >
> >>
> <https://gist. __githubusercontent.com/zcox/__ 3d6cf4329d49ca03271b/raw/ __57c46a5e4c9ea04d5c4209414d6f84__ 7492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
> >>
> >>
> <https://gist. __githubusercontent.com/zcox/__ 3d6cf4329d49ca03271b/raw/ __57c46a5e4c9ea04d5c4209414d6f84__ 7492d16c0d/gistfile1.txt
> <https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>>>
> >>
> >> >
> >> > >
> >> > > Seems like elasticsearch-hadoop tries talking to an ES node,
> >> it times out, tries the next one, it times out, etc until
> >> > > all nodes in the cluster are exhausted and then it gives up.
> >> > >
> >> > > As far as I can tell, the ES cluster is healthy while this is
> >> occurring. May map tasks are succeeding - probably about
> >> > > 10% of the attempts are killed due to this issue. The main
> >> problem is that these killed tasks waste a lot of time, and
> >> > > slow down the overall job execution.
> >> > >
> >> > > I'm not sure where to troubleshoot this next. Does anyone have
> >> any idea what would cause all of these time outs & failures?
> >> > >
> >> > > I'm also curious about the lines like this:
> >> > >
> >> > > 2014-09-30 12:49:20,469 WARN
> >> org.apache.commons.httpclient.__SimpleHttpConnectionManager:
> >> SimpleHttpConnectionManager being used incorrectly. Be sure that
> >> > HttpMethod.releaseConnection() is always called and that only
> >> one thread and/or method is using this connection
> >> > manager at a time.
> >> > >
> >> > >
> >> > > Would that be related to the timeout problem we're seeing?
> >> > >
> >> > > Thanks,
> >> > > Zach
> >> > >
> >> > > --
> >> > > You received this message because you are subscribed to the
> >> Google Groups "elasticsearch" group.
> >> > > To unsubscribe from this group and stop receiving emails from
> >> it, send an email to
> >> > >elasticsearc...@googlegroups.__com <mailto:elasticsearc...@googlegroups.com>
> <mailto:elasticsearc...@__googlegroups.com <mailto:elasticsearc...@googlegroups.com>> <javascript:>
> >> <mailto:elasticsearch+__unsubscribe@googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <mailto:elasticsearch%__2Bunsubscribe@googlegroups.com
> <mailto:elasticsearch%252Bunsubscribe@googlegroups.com>__> <javascript:>
> >> <javascript:>>.
> >> > > To view this discussion on the web visit
> >> >
> >>
> >https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>>
> >> >
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>>>
> >> > >
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>
> >>
> >> >
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/f304a286-__ 399f-4dea-b7f0-032b19ad67e6% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
> >>
> >> >
> >> > > For more options, visithttps://groups.google.__com/d/optout
> <http://groups.google.com/d/optout> <http://groups.google.com/d/__optout <http://groups.google.com/d/optout>>
> >> <http://groups.google.com/d/__optout <http://groups.google.com/d/optout>>
> <https://groups.google.com/d/__optout <https://groups.google.com/d/optout>
> >> <https://groups.google.com/d/__optout <https://groups.google.com/d/optout>>>.
> >> > --
> >> > Costin
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> Groups "elasticsearch" group.
> >> > To unsubscribe from this group and stop receiving emails from it,
> >> send an email to
> >> >elasticsearc...@googlegroups.__com <mailto:elasticsearc...@googlegroups.com>
> <mailto:elasticsearc...@__googlegroups.com <mailto:elasticsearc...@googlegroups.com>> <javascript:>
> >> <mailto:elasticsearch+__unsubscribe@googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <mailto:elasticsearch%__2Bunsubscribe@googlegroups.com
> <mailto:elasticsearch%252Bunsubscribe@googlegroups.com>__> <javascript:>>.
> >> > To view this discussion on the web visit
> >>
> >>
> >https://groups.google.com/d/ __msgid/elasticsearch/034d651e-__ 8562-4dde-bbb9-b3fef6d0d0b9%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/034d651e-__ 8562-4dde-bbb9-b3fef6d0d0b9%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com>>
> >> >
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/034d651e-__ 8562-4dde-bbb9-b3fef6d0d0b9% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/034d651e-__ 8562-4dde-bbb9-b3fef6d0d0b9% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
> >>
> >> > For more options, visithttps://groups.google.__com/d/optout
> <http://groups.google.com/d/optout> <http://groups.google.com/d/__optout <http://groups.google.com/d/optout>>
> >> <https://groups.google.com/d/__optout <https://groups.google.com/d/optout>>.
> >>
> >> --
> >> Costin
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "elasticsearch" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to
> >>elasticsearch+unsubscribe@__googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>
> <mailto:elasticsearch% __2Bunsubscribe@googlegroups.com <mailto:elasticsearch%252Bunsubscribe@googlegroups.com>__ >
> >> <mailto:elasticsearch+__unsubscribe@googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <mailto:elasticsearch%__2Bunsubscribe@googlegroups.com
> <mailto:elasticsearch%252Bunsubscribe@googlegroups.com>__>>.
> >> To view this discussion on the web visit
> >>
> 
> >>https://groups.google.com/d/ __msgid/elasticsearch/90b059ef-__ 680e-4b5c-a0c9-dc5e5038205a%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com>
> >>
> >>
> <https://groups.google.com/d/ __msgid/elasticsearch/90b059ef-__ 680e-4b5c-a0c9-dc5e5038205a% __40googlegroups.com?utm_medium=__ email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >> For more options, visithttps://groups.google.__com/d/optout <http://groups.google.com/d/optout>.
> >
> >
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> >https://groups.google.com/d/ __topic/elasticsearch/__ BKR18lczF1w/unsubscribe
> <https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe>.
> > To unsubscribe from this group and all its topics, send an email to
> >elasticsearch+unsubscribe@__googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>
> <mailto:elasticsearch% __2Bunsubscribe@googlegroups.com <mailto:elasticsearch%252Bunsubscribe@googlegroups.com>__ >.
> > To view this discussion on the web visit
> >https://groups.google.com/d/ __msgid/elasticsearch/542BF16B.__ 6010103%40gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com>.
> >
> > For more options, visithttps://groups.google.__com/d/optout <http://groups.google.com/d/optout>.
> 
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> elasticsearch+unsubscribe@__googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>
> <mailto:elasticsearch+__unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/ __msgid/elasticsearch/__ CANjo42zgBXS4Y%3DSoRG67Q% __3DeqH4ctdJDbOnzFSR6PjVZvQr-__ A8w%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com>
> <https://groups.google.com/d/ __msgid/elasticsearch/__ CANjo42zgBXS4Y%3DSoRG67Q% __3DeqH4ctdJDbOnzFSR6PjVZvQr-__ A8w%40mail.gmail.com?utm___medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com?utm_medium=email&utm_source=footer>__>.
> For more options, visit https://groups.google.com/d/__optout <https://groups.google.com/d/optout>.
> 
> --
> Costin
> 
> --
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/ __topic/elasticsearch/__ BKR18lczF1w/unsubscribe
> <https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@__googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/ __msgid/elasticsearch/542EC4BA.__ 60204%40gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/542EC4BA.60204%40gmail.com>.
> 
> For more options, visit https://groups.google.com/d/__optout <https://groups.google.com/d/optout>.
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-L2pT8MgtQA%40mail.gmail.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> --  
> Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/542ECE86.4090001%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/542ECE86.4090001%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Zach\_Cox](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_cox/32/1248_2.png) [@Zach\_Cox](https://discuss.elastic.co/u/Zach_Cox)\
**Post date:** [October 3, 2014, 5:11pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/13 "2014-10-03T17:11:54Z")

</div>

Our 4 ES nodes are all m1.large (  
[Amazon EC2 Instance Comparison](http://www.ec2instances.info/?filter=m1.large)) and our 5 Hadoop nodes are  
all m1.xlarge ([Amazon EC2 Instance Comparison](http://www.ec2instances.info/?filter=m1.xlarge)).

Thanks for the troubleshooting pointers - we'll do some more research.

On Fri, Oct 3, 2014 at 11:27 AM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:

> What type of AWS instances are you using? Virtualization tends to  
> interfere in various ways with a running system - sometime for good,  
> sometimes for worse.
> 
> The number of tasks is good to compute the total number of data and  
> entries you are throwing at ES at one time. You are looking at a maximum of  
> 25 MBs or 25K entries against the 4 node Elastic. Depending on how beefy  
> the machines are and what type of storage and mapping you have, it might be  
> quick or slow - we can only guess. Try using Marvel or other monitoring  
> tools so you can see whether there's load building up or if anything  
> unusual happens.
> 
> Since it's unclear what the issue might be, take baby steps [1] and start  
> with minimal load (smaller bulk size + less tasks) see whether there are  
> any issues and keep on going.
> 
> [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/)  
> master/troubleshooting.html
> 
> On 10/3/14 6:59 PM, Zach Cox wrote:
> 
> > Our Hadoop and Elasticsearch are all on AWS. We have 2 MR jobs that write  
> > to ES - 1 of them works fine, and one of them  
> > takes forever due to 10-20% of tasks failing in the way I've described.  
> > So I don't think it's any kind of  
> > network/firewall issue. There are no nightly backups related to ES or  
> > anything.
> > 
> > Would 25 map tasks batch writing to a 4-node Elasticsearch cluster cause  
> > it so much pain that it stops responding to  
> > writes? If I ssh to the ES nodes while this job is running and some tasks  
> > are failing, it doesn't seem like ES is under  
> > much stress.
> > 
> > Thanks,  
> > Zach
> > 
> > On Fri, Oct 3, 2014 at 10:46 AM, Costin Leau \<[costin.leau@gmail.com](mailto:costin.leau@gmail.com)  
> > [mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com)\> wrote:
> > 
> > ```
> > You can always enable TRACE though that is likely to create way too
> > 
> > ```
> > 
> > much information in production and slow things  
> > down considerably.
> > 
> > ```
> > The first thing you can do is minimize the batch size to give ES more
> > 
> > ```
> > 
> > breathing space by minimizing the batch size  
> > (say to 512KB) or the number  
> > of entries (500 instead of 1k, etc...).  
> > However the error indicates a network issue not an Elasticsearch one  
> > so check whether there's some type of  
> > service/network/firewall initialization happening at night. Do the  
> > errors occur around the same time? Is there some  
> > backup procedure that potentially kicks in?
> > 
> > ```
> > Potentially you can try and increase the default http timeout
> > 
> > ```
> > 
> > (es.http.timeout) from 1m to 3m or so. However this is  
> > really a patch since if the ES server doesn't return a response in  
> > 1m, it means things are not going well at all.
> > 
> > ```
> > On 10/3/14 6:09 PM, Zach Cox wrote:
> > 
> > Is there anything else we could try here to debug
> > 
> > ```
> > 
> > elasticsearch-hadoop being unable to write to Elasticsearch? We're  
> > still seeing the same number of these fails during the nightly  
> > batch runs even after switching to  
> > 2.0.2.BUILD-SNAPSHOT,  
> > and I don't see any additional lines from  
> > org.elasticsearch.hadoop.\_\_\_\_rest in the tasktracker logs after adding  
> > to our  
> > logback.xml.
> > 
> > ```
> > Here are example logs for a task that failed this morning:
> > https://gist. __githubusercontent.com/zcox/__
> > 
> > ```
> > 
> > c5c81f4f8ee26d7bedbf/raw/\_\_ac9bef7bfe3a8e1ef69a62aabe4c7c  
> > \_\_3983882f19/gistfile1.txt  
> > \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c  
> > 3983882f19/gistfile1.txt\>
> > 
> > ```
> > Is it reasonable to expect 4 Elasticsearch nodes to handle the
> > 
> > ```
> > 
> > batch write volume from 5 Hadoop nodes (25 map  
> > tasks)?
> > 
> > ```
> > Thanks,
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox <zcox522@gmail.com
> > 
> > ```
> > 
> > [mailto:zcox522@gmail.com](mailto:zcox522@gmail.com) \<[mailto:zcox522@gmail.com](mailto:zcox522@gmail.com)  
> > [mailto:zcox522@gmail.com](mailto:zcox522@gmail.com)\>\> wrote:
> > 
> > ```
> > Hi Costin - by "bulk size/entries number" are you referring
> > 
> > ```
> > 
> > to the es.batch.size.bytes and  
> > es.batch.size.entries  
> > config values described here?
> > 
> > ```
> > http://www.elasticsearch.org/__guide/en/elasticsearch/
> > 
> > ```
> > 
> > hadoop/\_\_master/configuration.html#\_\_configuration-serialization  
> > \<[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/)  
> > master/configuration.html#configuration-serialization\>
> > 
> > ```
> > It looks like the only elasticsearch-related config values
> > 
> > ```
> > 
> > we're setting is this:
> > 
> > ```
> > es.input.json = true
> > 
> > So we must be using default values for those es.batch.size
> > 
> > ```
> > 
> > config values.
> > 
> > ```
> > Thanks,
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <zcox522@gmail.com
> > 
> > ```
> > 
> > [mailto:zcox522@gmail.com](mailto:zcox522@gmail.com)  
> > \<[mailto:zcox522@gmail.com](mailto:zcox522@gmail.com) [mailto:zcox522@gmail.com](mailto:zcox522@gmail.com)\>\> wrote:
> > 
> > ```
> > This particular job has 1353 map tasks, Hadoop cluster
> > 
> > ```
> > 
> > has 5 nodes with total map task capacity of 25.  
> > Elasticsearch cluster has 4 nodes.
> > 
> > ```
> > Where can I find the bulk size/entries numbers?
> > 
> > Thanks,
> > Zach
> > 
> > On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <
> > 
> > ```
> > 
> > [costin.leau@gmail.com](mailto:costin.leau@gmail.com) [mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com)  
> > \<[mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com) [mailto:costin.leau@gmail.com](mailto:costin.leau@gmail.com)\>\_\_\>  
> > wrote:  
> > \> The error indicates the ES nodes don't reply in a  
> > timely fashion and thus  
> > \> the connection drops. Based on your logs it seems to  
> > be either a GC or a  
> > \> network issue.  
> > \> You could try turning on logging in package  
> > 'org.elasticsearch.hadoop.\_\_rest'
> > 
> > ```
> > > to DEBUG.
> > > How many tasks do you have and what's your bulk
> > 
> > ```
> > 
> > size/entries number?  
> > \>  
> > \>  
> > \> On 10/1/14 2:15 PM, Zach Cox wrote:  
> > \>\>  
> > \>\> Hi Costin - we updated our dependencies to use  
> > elasticsearch-hadoop  
> > \>\> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change  
> > \>\> anything. We're still seeing the same task failures  
> > while trying to write  
> > \>\> to Elasticsearch. The only difference in the  
> > \>\> logs is that now I don't see the  
> > SimpleHttpConnectionManager warnings.  
> > \>\>  
> > \>\> Any ideas what we could try next?  
> > \>\>  
> > \>\> Thanks,  
> > \>\> Zach  
> > \>\>  
> > \>\>  
> > \>\> On Tuesday, September 30, 2014 10:54:27 AM UTC-5,  
> > Costin Leau wrote:  
> > \>\>  
> > \>\> Can you please try the 2.0.2.BUILD-SNAPSHOT? I  
> > think you might be  
> > \>\> running into issue #256 which was fixed some time ago  
> > \>\> and will be part of the upcoming  
> > \>\> 2.0.2, 2.1 Beta2.  
> > \>\>  
> > \>\> Cheers,  
> > \>\>  
> > \>\> On 9/30/14 6:43 PM, Zach Cox wrote:  
> > \>\> \> Hi Costin:  
> > \>\> \>  
> > \>\> \> elasticsearch-hadoop 2.0.0  
> > \>\> \> cascading 2.5.4  
> > \>\> \> scalding 0.10.0  
> > \>\> \>  
> > \>\> \> Thanks,  
> > \>\> \> Zach  
> > \>\> \>  
> > \>\> \>  
> > \>\> \> On Tuesday, September 30, 2014 10:25:10 AM  
> > UTC-5, Costin Leau wrote:  
> > \>\> \>  
> > \>\> \> What version of es-hadoop/es/cascading are  
> > you using?  
> > \>\> \>  
> > \>\> \> On 9/30/14 6:16 PM, Zach Cox wrote:  
> > \>\> \> \> Hi - we're having problems with one of  
> > our map-reduce jobs  
> > \>\> that writes to Elasticsearch. Lots of map tasks are  
> > failing  
> > \>\> \> \> due to ES being "unavailable", with logs  
> > like this:  
> > \>\> \> \>  
> > \>\> \>  
> > \>\>  
> > \>[https://gist](https://gist).**[githubusercontent.com/zcox/](http://githubusercontent.com/zcox/)**  
> > 3d6cf4329d49ca03271b/raw/\_\_57c46a5e4c9ea04d5c4209414d6f84  
> > \_\_7492d16c0d/gistfile1.txt \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>  
> > \>\>  
> > \>\>  
> > \<[https://gist](https://gist).**[githubusercontent.com/zcox/](http://githubusercontent.com/zcox/)**  
> > 3d6cf4329d49ca03271b/raw/\_\_57c46a5e4c9ea04d5c4209414d6f84  
> > \_\_7492d16c0d/gistfile1.txt  
> > \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>\>  
> > \>\>  
> > \>\> \>  
> > \>\>  
> > \<[https://gist](https://gist).**[githubusercontent.com/zcox/](http://githubusercontent.com/zcox/)**  
> > 3d6cf4329d49ca03271b/raw/\_\_57c46a5e4c9ea04d5c4209414d6f84  
> > \_\_7492d16c0d/gistfile1.txt  
> > \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>  
> > \>\>  
> > \>\>  
> > \<[https://gist](https://gist).**[githubusercontent.com/zcox/](http://githubusercontent.com/zcox/)**  
> > 3d6cf4329d49ca03271b/raw/\_\_57c46a5e4c9ea04d5c4209414d6f84  
> > \_\_7492d16c0d/gistfile1.txt  
> > \<[zcox’s gists · GitHub](https://gist.githubusercontent.com/zcox/)  
> > 3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f84  
> > 7492d16c0d/gistfile1.txt\>\>\>  
> > \>\>  
> > \>\> \>  
> > \>\> \> \>  
> > \>\> \> \> Seems like elasticsearch-hadoop tries  
> > talking to an ES node,  
> > \>\> it times out, tries the next one, it times out, etc  
> > until  
> > \>\> \> \> all nodes in the cluster are exhausted  
> > and then it gives up.  
> > \>\> \> \>  
> > \>\> \> \> As far as I can tell, the ES cluster is  
> > healthy while this is  
> > \>\> occurring. May map tasks are succeeding - probably  
> > about  
> > \>\> \> \> 10% of the attempts are killed due to  
> > this issue. The main  
> > \>\> problem is that these killed tasks waste a lot of  
> > time, and  
> > \>\> \> \> slow down the overall job execution.  
> > \>\> \> \>  
> > \>\> \> \> I'm not sure where to troubleshoot this  
> > next. Does anyone have  
> > \>\> any idea what would cause all of these time outs &  
> > failures?  
> > \>\> \> \>  
> > \>\> \> \> I'm also curious about the lines like  
> > this:  
> > \>\> \> \>  
> > \>\> \> \> 2014-09-30 12:49:20,469 WARN  
> > \>\> org.apache.commons.httpclient.  
> > \_\_SimpleHttpConnectionManager:  
> > \>\> SimpleHttpConnectionManager being used incorrectly.  
> > Be sure that  
> > \>\> \> HttpMethod.releaseConnection() is always  
> > called and that only  
> > \>\> one thread and/or method is using this connection  
> > \>\> \> manager at a time.  
> > \>\> \> \>  
> > \>\> \> \>  
> > \>\> \> \> Would that be related to the timeout  
> > problem we're seeing?  
> > \>\> \> \>  
> > \>\> \> \> Thanks,  
> > \>\> \> \> Zach  
> > \>\> \> \>  
> > \>\> \> \> --  
> > \>\> \> \> You received this message because you are  
> > subscribed to the  
> > \>\> Google Groups "elasticsearch" group.  
> > \>\> \> \> To unsubscribe from this group and stop  
> > receiving emails from  
> > \>\> it, send an email to  
> > \>\> \> \>elasticsearc...@googlegroups.\_\_com  
> > [mailto:elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)  
> > \<mailto:elasticsearc...@\_\_googlegroups.com \<mailto:  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\>\> \<javascript:\>  
> > \>\> \<[mailto:elasticsearch+\_\_unsubscribe@googlegroups.com](mailto:elasticsearch+__unsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%**[2Bunsubscribe@googlegroups.com](mailto:2Bunsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%252Bunsubscribe@googlegroups.com](mailto:elasticsearch%252Bunsubscribe@googlegroups.com)**\>  
> > \<javascript:\>  
> > \>\> \<javascript:\>\>.  
> > \>\> \> \> To view this discussion on the web visit  
> > \>\> \>  
> > \>\>  
> > \>[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>\>  
> > \>\> \>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40googlegroups.com](http://40googlegroups.com)\>\>\>  
> > \>\> \> \>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>  
> > \>\>  
> > \>\> \>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/f304a286-](https://groups.google.com/d/__msgid/elasticsearch/f304a286-)  
> > \_\_399f-4dea-b7f0-032b19ad67e6%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/f304a286-](https://groups.google.com/d/msgid/elasticsearch/f304a286-)  
> > 399f-4dea-b7f0-032b19ad67e6%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>\>\>.  
> > \>\>  
> > \>\> \>  
> > \>\> \> \> For more options,  
> > visithttps://groups.google.\_\_com/d/optout  
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<  
> > [http://groups.google.com/d/\_\_optout](http://groups.google.com/d/__optout) [http://groups.google.com/d/optout](http://groups.google.com/d/optout)\>  
> > \>\> \<[http://groups.google.com/d/\_\_optout](http://groups.google.com/d/__optout) \<  
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout)\>\>  
> > \<[https://groups.google.com/d/\_\_optout](https://groups.google.com/d/__optout) \<  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>  
> > \>\> \<[https://groups.google.com/d/\_\_optout](https://groups.google.com/d/__optout) \<  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>\>\>.  
> > \>\> \> --  
> > \>\> \> Costin  
> > \>\> \>  
> > \>\> \> --  
> > \>\> \> You received this message because you are  
> > subscribed to the Google  
> > \>\> Groups "elasticsearch" group.  
> > \>\> \> To unsubscribe from this group and stop  
> > receiving emails from it,  
> > \>\> send an email to  
> > \>\> \>elasticsearc...@googlegroups.\_\_com \<mailto:  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\>  
> > \<mailto:elasticsearc...@\_\_googlegroups.com \<mailto:  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\>\> \<javascript:\>  
> > \>\> \<[mailto:elasticsearch+\_\_unsubscribe@googlegroups.com](mailto:elasticsearch+__unsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%**[2Bunsubscribe@googlegroups.com](mailto:2Bunsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%252Bunsubscribe@googlegroups.com](mailto:elasticsearch%252Bunsubscribe@googlegroups.com)**\>  
> > \<javascript:\>\>.  
> > \>\> \> To view this discussion on the web visit  
> > \>\>  
> > \>\>  
> > \>[https://groups.google.com/d/\_\_msgid/elasticsearch/034d651e-](https://groups.google.com/d/__msgid/elasticsearch/034d651e-)  
> > \_\_8562-4dde-bbb9-b3fef6d0d0b9%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/034d651e-](https://groups.google.com/d/__msgid/elasticsearch/034d651e-)  
> > \_\_8562-4dde-bbb9-b3fef6d0d0b9%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40googlegroups.com](http://40googlegroups.com)\>\>  
> > \>\> \>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/034d651e-](https://groups.google.com/d/__msgid/elasticsearch/034d651e-)  
> > \_\_8562-4dde-bbb9-b3fef6d0d0b9%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/034d651e-](https://groups.google.com/d/__msgid/elasticsearch/034d651e-)  
> > \_\_8562-4dde-bbb9-b3fef6d0d0b9%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=**email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/034d651e-](https://groups.google.com/d/msgid/elasticsearch/034d651e-)  
> > 8562-4dde-bbb9-b3fef6d0d0b9%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>\>.  
> > \>\>  
> > \>\> \> For more options, visithttps://groups.google.**  
> > com/d/optout  
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout) \<  
> > [http://groups.google.com/d/\_\_optout](http://groups.google.com/d/__optout) [http://groups.google.com/d/optout](http://groups.google.com/d/optout)\>  
> > \>\> \<[https://groups.google.com/d/\_\_optout](https://groups.google.com/d/__optout) \<  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>\>.  
> > \>\>  
> > \>\> --  
> > \>\> Costin  
> > \>\>  
> > \>\> --  
> > \>\> You received this message because you are subscribed  
> > to the Google Groups  
> > \>\> "elasticsearch" group.  
> > \>\> To unsubscribe from this group and stop receiving  
> > emails from it, send an  
> > \>\> email to  
> > \>\>elasticsearch+unsubscribe@\_\_googlegroups.com \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>  
> > \<mailto:elasticsearch%**[2Bunsubscribe@googlegroups.com](mailto:2Bunsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%252Bunsubscribe@googlegroups.com\>**\>  
> > \>\> \<[mailto:elasticsearch+\_\_unsubscribe@googlegroups.com](mailto:elasticsearch+__unsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%**[2Bunsubscribe@googlegroups.com](mailto:2Bunsubscribe@googlegroups.com)  
> > [mailto:elasticsearch%252Bunsubscribe@googlegroups.com](mailto:elasticsearch%252Bunsubscribe@googlegroups.com)**\>\>.  
> > \>\> To view this discussion on the web visit  
> > \>\>
> > 
> > ```
> > >>https://groups.google.com/d/__msgid/elasticsearch/
> > 
> > ```
> > 
> > 90b059ef-\_\_680e-4b5c-a0c9-dc5e5038205a%\_\_40googlegroups.com  
> > \<[https://groups.google.com/d/msgid/elasticsearch/90b059ef-](https://groups.google.com/d/msgid/elasticsearch/90b059ef-)  
> > 680e-4b5c-a0c9-dc5e5038205a%[40googlegroups.com](http://40googlegroups.com)\>  
> > \>\>  
> > \>\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/90b059ef-](https://groups.google.com/d/__msgid/elasticsearch/90b059ef-)  
> > \_\_680e-4b5c-a0c9-dc5e5038205a%\__[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm)_  
> > medium=\_\_email&utm\_source=footer  
> > \<[https://groups.google.com/d/msgid/elasticsearch/90b059ef-](https://groups.google.com/d/msgid/elasticsearch/90b059ef-)  
> > 680e-4b5c-a0c9-dc5e5038205a%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>\>.  
> > \>\> For more options, visithttps://groups.google.\_\_com/d/optout  
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout).  
> > \>  
> > \>  
> > \> --  
> > \> Costin  
> > \>  
> > \> --  
> > \> You received this message because you are subscribed  
> > to a topic in the  
> > \> Google Groups "elasticsearch" group.  
> > \> To unsubscribe from this topic, visit  
> > \>[https://groups.google.com/d/\_\_topic/elasticsearch/\_\_](https://groups.google.com/d/ __topic/elasticsearch/__ )  
> > BKR18lczF1w/unsubscribe  
> > \<[https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > BKR18lczF1w/unsubscribe\>.  
> > \> To unsubscribe from this group and all its topics,  
> > send an email to  
> > \>elasticsearch+unsubscribe@\_\_googlegroups.com \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>  
> > \<mailto:elasticsearch%**[2Bunsubscribe@googlegroups.com](mailto:2Bunsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%252Bunsubscribe@googlegroups.com\>**\>.  
> > \> To view this discussion on the web visit  
> > \>[https://groups.google.com/d/\_](https://groups.google.com/d/_)  
> > \_msgid/elasticsearch/542BF16B.\_\_6010103%[40gmail.com](http://40gmail.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/542BF16B](https://groups.google.com/d/msgid/elasticsearch/542BF16B).  
> > 6010103%[40gmail.com](http://40gmail.com)\>.  
> > \>  
> > \> For more options, visithttps://groups.google.\_\_com/d/optout  
> > [http://groups.google.com/d/optout](http://groups.google.com/d/optout).
> > 
> > ```
> > --
> > You received this message because you are subscribed to the
> > 
> > ```
> > 
> > Google Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it,  
> > send an email to  
> > elasticsearch+unsubscribe@\_\_googlegroups.com \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>  
> > \<mailto:elasticsearch+**[unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com) \<mailto:  
> > elasticsearch%2Bunsubscribe@googlegroups.com\>\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/\_\_msgid/elasticsearch/\_\_](https://groups.google.com/d/ __msgid/elasticsearch/__ )  
> > CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-  
> > A8w%[40mail.gmail.com](http://40mail.gmail.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%[40mail.gmail.com](http://40mail.gmail.com)\>  
> > \<[https://groups.google.com/d/\_\_msgid/elasticsearch/\_\_](https://groups.google.com/d/ __msgid/elasticsearch/__ )  
> > CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-  
> > A8w%[40mail.gmail.com?utm\_\_\_medium=email&utm\_source=footer](http://40mail.gmail.com?utm___medium=email&utm_source=footer)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-  
> > A8w%[40mail.gmail.com?utm\_medium=email&utm\_source=footer](http://40mail.gmail.com?utm_medium=email&utm_source=footer)\>**\>.  
> > For more options, visit [https://groups.google.com/d/\_\_optout](https://groups.google.com/d/__optout) \<  
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.
> > 
> > ```
> > --
> > Costin
> > 
> > --
> > You received this message because you are subscribed to a topic in
> > 
> > ```
> > 
> > the Google Groups "elasticsearch" group.  
> > To unsubscribe from this topic, visit [https://groups.google.com/d/\_\_](https://groups.google.com/d/__)  
> > topic/elasticsearch/\_\_BKR18lczF1w/unsubscribe  
> > \<[https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > BKR18lczF1w/unsubscribe\>.  
> > To unsubscribe from this group and all its topics, send an email to  
> > elasticsearch+unsubscribe@\_\_googlegroups.com  
> > [mailto:elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/\_\_msgid/elasticsearch/542EC4BA.\_](https://groups.google.com/d/__msgid/elasticsearch/542EC4BA._)  
> > \_60204%[40gmail.com](http://40gmail.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/542EC4BA](https://groups.google.com/d/msgid/elasticsearch/542EC4BA).  
> > 60204%[40gmail.com](http://40gmail.com)\>.
> > 
> > ```
> > For more options, visit https://groups.google.com/d/__optout <
> > 
> > ```
> > 
> > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:elasticsearch+  
> > [unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com)\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-  
> > L2pT8MgtQA%[40mail.gmail.com](http://40mail.gmail.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > CANjo42yKPnbH6R%3DrzM%3DrRBo%3DUqcD%2BbOy8%3D8y5Xo-  
> > L2pT8MgtQA%[40mail.gmail.com?utm\_medium=email&utm\_source=footer](http://40mail.gmail.com?utm_medium=email&utm_source=footer)\>.  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> topic/elasticsearch/BKR18lczF1w/unsubscribe.  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> msgid/elasticsearch/542ECE86.4090001%[40gmail.com](http://40gmail.com).
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANjo42z%2BJau8LFcOROa15xN2nOUzyr7PqK9Ky%2BqkkCN-6JjkbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANjo42z%2BJau8LFcOROa15xN2nOUzyr7PqK9Ky%2BqkkCN-6JjkbQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:58am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-sporadic-timeouts/20014/14 "2017-07-06T00:58:23Z")

</div>