# Dropping Documents on Hive Import with elasticsearch-hadoop-5.4.0

**URL:** <https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 18, 2017, 7:01pm UTC](https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325 "2017-05-18T19:01:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kpaAZ](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kpaAZ](https://discuss.elastic.co/u/kpaAZ)\
**Post date:** [May 18, 2017, 7:01pm UTC](https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325/1 "2017-05-18T19:01:49Z")

</div>

We've got a fairly simple Hive pipeline setup for basic reporting. Just recently updated the ES adapter and ES cluster.

We've been trying to run an import job from ES (5.3.0, 3 data nodes) to Hive (Hive 1.1.0-cdh5.7.0) using the elasticsearch-hadoop-5.4.0 adapter but have noticed that multiple documents are missing. Previously, we were using version 2.3.2 of the adapter with version 1.7 of ES (also 3 data nodes).

```
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = '${esIndex}', 
   'es.nodes' = '${esHost}',
   'es.scroll.size' = '2000',
   'es.mapping.names' = '${esMappings}',
   'es.nodes.wan.only' = 'false',
   'es.query' = '?q=${esQuery}');

```

I've been hitting our ES instance directly using the query for a small subset of the import and get all expected documents. When I do a simple select query against the table, I seem to be missing between 1-10 documents (expected 32).

`ADD JAR hdfs:///path/to/jars/elasticsearch-hadoop-5.4.0.jar;`

select \* from myDB.myTable where ;  
Additionally, the documents which are missing are not consistent. If I run the hive query multiple times, different documents will be missing each time.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [May 20, 2017, 6:46pm UTC](https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325/2 "2017-05-20T18:46:00Z")

</div>

Could you enable `TRACE` level logging for the `org.elasticsearch.hadoop.rest.commonshttp` package? This should bring up each request to and response from Elasticsearch that the connector sends. After doing so, could you check to see if all the expected documents are making it back across the wire in the logs? This will highlight if it is a problem with the connector or with Elasticsearch.

---

<div class="post-metadata">

**Author:** ![kpaAZ](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kpaAZ](https://discuss.elastic.co/u/kpaAZ)\
**Post date:** [May 31, 2017, 8:37pm UTC](https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325/3 "2017-05-31T20:37:40Z")

</div>

We've gone ahead and set the logging level to `TRACE` for that package. Unfortunately, for the import itself, we are pulling in approximately 3.8 million documents into a secondary parquet table which I am then querying making it difficult to verify what was dropped.

Something that I have noticed and seems to have corrected my issue is the adjustment of  
`es.input.max.docs.per.partition`. I changed this from the default of `100000` to `500000` and no longer seem to have any issues with dropped documents so far. I'm relatively new to this adapter/ES and not entirely sure why this seems to have resolved the issue. Additionally, if this is the correct fix, I'm curious about what might happen in the future as the number of records to import grows.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2017, 8:38pm UTC](https://discuss.elastic.co/t/dropping-documents-on-hive-import-with-elasticsearch-hadoop-5-4-0/86325/4 "2017-06-28T20:38:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.