# Hive to Elastic Data Load **URL:** \ **Category:** Elasticsearch\ **Tags:** docker, es-hadoop\ **Created:** [April 15, 2020, 9:30am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066 "2020-04-15T09:30:19Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 15, 2020, 9:30am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/1 "2020-04-15T09:30:20Z")
Hi, I have loaded a data containing **536965213** rows from hive to elastic. But when I check the data loaded in elastic via Kibana it displays " **548,889,213** hits which is more than the number of records present in my hive table. I had read about "precision\_threshold" from the below link. [https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html) But I still have few questions: - If the data cardinality is high then why does elastic create some extra records which might become a memory issue where there is an use case like this i.e., dealing with huge data? - Will this produce any bad/duplicate data when I try reading data using Java REST API? Also in the link provided above explains me to add hash properties in my mapping. - Does this mean that do I need to add hash property to all the fields that I use in my search usecase? or is it only for making my search more quicker? - Can this count in my elastic search matches with my hive after using this "hash" property in mapping? Just for reference. Below are the table properties used and my table has **40** columns as of now and they might increase till **150** columns moving forward. ``` ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1') LOCATION 'maprfs:/XXX/xxxx/XXXX.db/xxxxxx' TBLPROPERTIES ( 'es.index.auto.create'='true', 'es.index.read.missing.as.empty'='true', 'es.net.ssl'='false', 'es.nodes'='http://xxxxxxx', 'es.nodes.client.only'='false', 'es.nodes.discovery'='true', 'es.nodes.wan.only'='false', 'es.port'='9200', 'es.resource'='eshive/xxxx', 'last_modified_by'='xxxxx', 'last_modified_time'='1584455649', 'numFiles'='0', 'numRows'='0', 'rawDataSize'='0', 'totalSize'='0', 'transient_lastDdlTime'='1584455649') ``` Regards, Sarath ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 16, 2020, 7:50am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/2 "2020-04-16T07:50:26Z")
@dadoonet can you please help me on this? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 16, 2020, 8:18am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/3 "2020-04-16T08:18:30Z")
Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the [Community Code of Conduct](https://www.elastic.co/community/codeofconduct) specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you. If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a [subscription](https://www.elastic.co/subscriptions). It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 16, 2020, 8:19am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/4 "2020-04-16T08:19:01Z")
I don't know Hive. What is the index name? ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 16, 2020, 8:53am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/5 "2020-04-16T08:53:18Z")
Index name is "eshive". ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 16, 2020, 8:56am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/6 "2020-04-16T08:56:50Z")
My Apologies for tagging you. I really don't know that this is something that was addressed previously in this forum. Please share link if this is addressed previously. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 16, 2020, 9:06am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/7 "2020-04-16T09:06:31Z")
How do you check the number of documents in Kibana? Are you using a cardinality aggregation given that you linked to this or looking at e.g. the output of `_cat/indices`? ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 16, 2020, 9:16am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/8 "2020-04-16T09:16:13Z")
I am simply navigating to discover tab and checking the number of documents present under that particular index. ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 17, 2020, 12:21pm UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/9 "2020-04-17T12:21:03Z")
> [@Christian\_Dahlqvist](#): > > `_cat/indices` ? This is the output I got when I execute \_cat/indices: `green open eshive_xxxx PPwGbKyKSpujbZDbYM5jgw 1 1 548889213 0 321.5gb 145.7gb` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 17, 2020, 12:37pm UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/10 "2020-04-17T12:37:16Z")
It does seem like you have had some duplicates inserted as the document count is higher than what is expected, which will hve an impact on query results. It also looks like you have indexed all your data into a single primary shard which quite large. Clients indexing into Elasticsearch generally provide an at-least-once delivery guarantee, so if you push the cluster too hard or requests end up timing out, e.g. due to long GC it is possible that clients will resent the requests which can cause duplicates unless you specify the document ID client side. I do not know the ES-Hadoop plugin so am not sure how you control concurrency. I would recommend looking at the Elasticsearch logs around the period you performed the indexing and see if there is any evidence of long or slow GC or any other errors/issues. ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [April 21, 2020, 4:45pm UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/11 "2020-04-21T16:45:06Z")
Also worth investigating is if any of the write tasks failed and restarted on the Hadoop side. If any tasks fail when writing data to Elasticsearch they will be restarted, and all previously accepted documents from that split will be rewritten. This can be avoided by telling ES-Hadoop which field should be considered a unique identifier for a document. In this case, ES-Hadoop will make sure the value from that field is extracted and set as the document's id for the purposes of avoiding duplicates. ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 22, 2020, 10:00am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/12 "2020-04-22T10:00:17Z")
Sure James I will investigate on this and get back to you with the details. ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 24, 2020, 2:18pm UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/13 "2020-04-24T14:18:15Z")
Yes, this count is because of the mappers that were killed. Thankq for bringing this point. ---
**Author:** ![Sarath\_Pullabhotla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarath_pullabhotla/32/65563_2.png) [@Sarath\_Pullabhotla](https://discuss.elastic.co/u/Sarath_Pullabhotla)\ **Post date:** [April 27, 2020, 11:23am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/14 "2020-04-27T11:23:08Z")
Is there any way we can restrict the duplicate records in this scenario? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 25, 2020, 11:23am UTC](https://discuss.elastic.co/t/hive-to-elastic-data-load/228066/15 "2020-05-25T11:23:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.