# Why my master node receives many POST bulk requests when using es-hive-connector to create index?

**URL:** <https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [March 31, 2017, 9:28am UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790 "2017-03-31T09:28:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dillon\_Peng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dillon_peng/32/16791_2.png) [@Dillon\_Peng](https://discuss.elastic.co/u/Dillon_Peng)\
**Post date:** [March 31, 2017, 9:28am UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790/1 "2017-03-31T09:28:22Z")

</div>

My hive has more than 30 nodes, and my table's space is almost 140GB, another, my elasticsearch cluster ( 3 data nodes with 8 cores/16G memory) is isolated from the hive. Now,  
I want to load data from hive into es according Apache Hive integration.

The following is my hiveQL script:

```
add jar elasticsearch-hadoop-5.2.2.jar;
drop table database_X.artists;
CREATE EXTERNAL TABLE database_X.artists(
user_id string, 
province int ,
...
col34 string) -- the table has 34 columns
stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
tblproperties('es.resource' = 'dillon_pengcz/artists', 'es.nodes' = '172.21.8.24', 'es.index.auto.create' = 'true', 'es.mapping.id'='caa_id', 'es.batch.size.entries'='0', 'es.batch.size.bytes' = '4mb');
insert overwrite table database_X.artists select * from database_X.artists_src;

```

'172.21.8.24' is my ES master node ip

These days I can not successfully executed the above script. So I successfully tested `100000` records through _limit_ as follows:  
`insert overwrite table database_X.artists select * from database_X.artists_src limit 100000;`  
And I used `tcpflow -p -c -i eth1 port 9200` to find what's happening. But I found something different from my understanding:

In my master node `172.21.8.24`, I got many many POST bulk request as follows:  
172.021.008.024.56340-172.021.008.034.09200: POST /\_bulk HTTP/1.1^M  
User-Agent: curl/7.19.7 (x86\_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2^M  
Host: 172.21.8.34:9200^M  
Accept: _/_^M  
Content-Length: 17500403^M  
Content-Type: application/x-www-form-urlencoded^M  
Expect: 100-continue^M  
^M

```
172.021.008.024.09200-172.021.008.034.56340: HTTP/1.1 100 Continue^M
^M
```

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [March 31, 2017, 6:34pm UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790/2 "2017-03-31T18:34:38Z")

</div>

What is the error message? From skimming this problem it looks like 140GB of data all at once is too much to handle.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [April 3, 2017, 3:28pm UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790/3 "2017-04-03T15:28:22Z")

</div>

@Dillon_Peng Could you include the trace level logs for the `org.elasticsearch.hadoop.rest.commonshttp` package and include them here? I imagine there may be something amiss with your discovery settings. Are you hosting your master node as a standalone master or is it also acting as a datanode?

---

<div class="post-metadata">

**Author:** ![Dillon\_Peng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dillon_peng/32/16791_2.png) [@Dillon\_Peng](https://discuss.elastic.co/u/Dillon_Peng)\
**Post date:** [April 5, 2017, 4:10am UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790/4 "2017-04-05T04:10:31Z")

</div>

hi, James and Jimmy  
I am so sorry for not replying in time! Last three days were my little holiday!  
I finally found that this result was relative to my wrong settings because, for testing performance of ES-hive, I copied the whole directory _elasticsearch-5.2.2_ into my testing and separated cluster(Of course I modified some obvious setting such as ips for original cluster). Later on, I built a new testing cluster from scratch with elasticsearch-5.2.2.tar.gz, the strange _POST_ disappeared.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2017, 4:10am UTC](https://discuss.elastic.co/t/why-my-master-node-receives-many-post-bulk-requests-when-using-es-hive-connector-to-create-index/80790/5 "2017-05-03T04:10:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.