Connecting hdfs and elasticsearch

[impu.vasudev]

Hi all,

I am new to Elasticsearch and kibana and I want to use Elasticsearch and kibana for one my project Real Time Monitoring,
I want establish two-way connection between hdfs and elasticsearch.
I used Hive external tables but its not working.
I am not finding a way , I am not getting how to monitor real time.
Can anyone please suggest me which method is better and how to implement.

Thank you

[Wayne_Taylor]

Hi @impu.vasudev - what steps did you follow?

[impu.vasudev]

1. I created internal table and loaded .csv file
   CREATE TABLE source(
   POLID INT, NAME STRING )
   ROW FORMAT DELIMITED FIELDS TERMINATED by ','
   STORED AS TEXTFILE
   LOCATION '/user/impu/csv/'

2. Created External table

CREATE EXTERNAL TABLE source_test(
POLID INT, NAME STRING )
stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES ('es.resource' = 'test/line',
'es.index.auto.create' = 'TRUE',
'es.nodes' = '10.222.x.x',
'es.port' = '9200',
'es.net.http.auth.user'='elastic',
'es.net.http.auth.pass'='changeme',
'es.nodes.discovery'='true',
'es.nodes.wan.only' = 'false')

3. INSERT OVERWRITE TABLE source_test
   SELECT POLID, NAME FROM source

4. select * from source_test
   Query did not fetch any data and after few mins i got session time out.
   Index was not created in ES.

[Wayne_Taylor]

Just obvious question - have you tried a curl command verifying connectivity to the cluster with the parameters you're passing?

[impu.vasudev]

I am running these queries in cloudera and using curl command i just checked the connectivity to that node.. and yes it is pinging .
one more question - In which path should i load jar file ?

[Wayne_Taylor]

Not sure if you are using local disk of HDFS. But I just did a brew install of hive and then just added the JAR dependency locally like:

add jar file:///Users/wtaylor/Downloads/elasticsearch-hadoop-6.3.0.jar

[impu.vasudev]

Only in this path or should I place the jar file in all. The nodes of cloudera
My ES is installed in cloudera

[Wayne_Taylor]

JAR file should be accessible on the node or a path that Hive can access.

[impu.vasudev]

I have added in hive class path but still my cloudera is not accessing jar file.
More over i am still unable to find why my select query is not working?

[Wayne_Taylor]

Well if you can't access JAR your query won't work.

[impu.vasudev]

I am adding jar file using,
Add Jar hdfs:///user/impu/es/elasticsearch-hadoop-6.3.0.jar
command but i have to add jar for every session, so i want to add in class path.