# Connecting hdfs and elasticsearch **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [July 6, 2018, 6:22am UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829 "2018-07-06T06:22:30Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 6:22am UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/1 "2018-07-06T06:22:30Z")
Hi all, I am new to Elasticsearch and kibana and I want to use Elasticsearch and kibana for one my project Real Time Monitoring, I want establish two-way connection between hdfs and elasticsearch. I used Hive external tables but its not working. I am not finding a way , I am not getting how to monitor real time. Can anyone please suggest me which method is better and how to implement. Thank you ---
**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Post date:** [July 6, 2018, 11:43am UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/2 "2018-07-06T11:43:55Z")
Hi @impu.vasudev - what steps did you follow? ---
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 11:52am UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/3 "2018-07-06T11:52:35Z")
1. I created internal table and loaded .csv file CREATE TABLE source( POLID INT, NAME STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED by ',' STORED AS TEXTFILE LOCATION '/user/impu/csv/' 2. Created External table CREATE EXTERNAL TABLE source\_test( POLID INT, NAME STRING ) stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ('es.resource' = 'test/line', 'es.index.auto.create' = 'TRUE', 'es.nodes' = '10.222.x.x', 'es.port' = '9200', 'es.net.http.auth.user'='elastic', 'es.net.http.auth.pass'='changeme', 'es.nodes.discovery'='true', 'es.nodes.wan.only' = 'false') 1. INSERT OVERWRITE TABLE source\_test SELECT POLID, NAME FROM source 2. select \* from source\_test Query did not fetch any data and after few mins i got session time out. Index was not created in ES. ---
**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Post date:** [July 6, 2018, 12:21pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/4 "2018-07-06T12:21:01Z")
Just obvious question - have you tried a curl command verifying connectivity to the cluster with the parameters you're passing? ---
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 12:28pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/5 "2018-07-06T12:28:05Z")
I am running these queries in cloudera and using curl command i just checked the connectivity to that node.. and yes it is pinging . one more question - In which path should i load jar file ? ---
**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Post date:** [July 6, 2018, 12:31pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/6 "2018-07-06T12:31:57Z")
Not sure if you are using local disk of HDFS. But I just did a brew install of hive and then just added the JAR dependency locally like: `add jar file:///Users/wtaylor/Downloads/elasticsearch-hadoop-6.3.0.jar` ---
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 12:32pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/7 "2018-07-06T12:32:25Z")
Only in this path or should I place the jar file in all. The nodes of cloudera My ES is installed in cloudera ---
**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Post date:** [July 6, 2018, 12:36pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/8 "2018-07-06T12:36:39Z")
JAR file should be accessible on the node or a path that Hive can access. ---
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 12:40pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/9 "2018-07-06T12:40:59Z")
I have added in hive class path but still my cloudera is not accessing jar file. More over i am still unable to find why my select query is not working? ---
**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Post date:** [July 6, 2018, 12:42pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/10 "2018-07-06T12:42:39Z")
Well if you can't access JAR your query won't work. ---
**Author:** ![impu.vasudev](https://avatars.discourse-cdn.com/v4/letter/i/e274bd/32.png) [@impu.vasudev](https://discuss.elastic.co/u/impu.vasudev)\ **Post date:** [July 6, 2018, 12:48pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/11 "2018-07-06T12:48:53Z")
I am adding jar file using, Add Jar hdfs:///user/impu/es/elasticsearch-hadoop-6.3.0.jar command but i have to add jar for every session, so i want to add in class path. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 3, 2018, 12:49pm UTC](https://discuss.elastic.co/t/connecting-hdfs-and-elasticsearch/138829/12 "2018-08-03T12:49:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.