# Throttling write speed in ES-Hadoop Connector

**URL:** <https://discuss.elastic.co/t/throttling-write-speed-in-es-hadoop-connector/62528>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [October 7, 2016, 8:10pm UTC](https://discuss.elastic.co/t/throttling-write-speed-in-es-hadoop-connector/62528 "2016-10-07T20:10:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kamaldeep\_Singh](https://avatars.discourse-cdn.com/v4/letter/k/cdc98d/32.png) [@Kamaldeep\_Singh](https://discuss.elastic.co/u/Kamaldeep_Singh)\
**Post date:** [October 7, 2016, 8:10pm UTC](https://discuss.elastic.co/t/throttling-write-speed-in-es-hadoop-connector/62528/1 "2016-10-07T20:10:58Z")

</div>

Hello,

I was working on ES - hadoop connector and I see that if you server has less memory writes keep on getting dropped.  
org.elasticsearch.hadoop.EsHadoopException: Could not write all entries (maybe ES was overloaded?). Bailing out...

As mentioned in [Pushback to hadoop from es on bulk load](https://discuss.elastic.co/t/pushback-to-hadoop-from-es-on-bulk-load/1535/5) there's no bi-directional communication between Hadoop and the connector - the connector cannot say, there's too much data, slow down.

Does anyone think it might be a good idea to use sth. like Blocking Queues here and add acks while writing (kafka 101) 🙂 so as to let the consumer (thread on ES) read at slower pace.

Else we would have to tune the batch size, write speed, http timeouts ourself.

I'm open to building/contributing to this its a good idea.

---

<div class="post-metadata">

**Author:** ![zhifengMaBeijing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zhifengmabeijing/32/23882_2.png) [@zhifengMaBeijing](https://discuss.elastic.co/u/zhifengMaBeijing)\
**Post date:** [November 6, 2016, 11:57pm UTC](https://discuss.elastic.co/t/throttling-write-speed-in-es-hadoop-connector/62528/2 "2016-11-06T23:57:51Z")

</div>

I like the idea!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:22pm UTC](https://discuss.elastic.co/t/throttling-write-speed-in-es-hadoop-connector/62528/3 "2017-07-06T13:22:41Z")

</div>


