Connecting hdfs and elasticsearch

Elastic Stack Elasticsearch
1

Hi all,

I am new to Elasticsearch and kibana and I want to use Elasticsearch and kibana for one my project Real Time Monitoring,
I want establish two-way connection between hdfs and elasticsearch.
I used Hive external tables but its not working.
I am not finding a way , I am not getting how to monitor real time.
Can anyone please suggest me which method is better and how to implement.

Thank you

2

Hi @impu.vasudev - what steps did you follow?

3

  1. I created internal table and loaded .csv file
    CREATE TABLE source(
    POLID INT, NAME STRING )
    ROW FORMAT DELIMITED FIELDS TERMINATED by ','
    STORED AS TEXTFILE
    LOCATION '/user/impu/csv/'

  2. Created External table

CREATE EXTERNAL TABLE source_test(
POLID INT, NAME STRING )
stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES ('es.resource' = 'test/line',
'es.index.auto.create' = 'TRUE',
'es.nodes' = '10.222.x.x',
'es.port' = '9200',
'es.net.http.auth.user'='elastic',
'es.net.http.auth.pass'='changeme',
'es.nodes.discovery'='true',
'es.nodes.wan.only' = 'false')

  1. INSERT OVERWRITE TABLE source_test
    SELECT POLID, NAME FROM source

  2. select * from source_test
    Query did not fetch any data and after few mins i got session time out.
    Index was not created in ES.

4

Just obvious question - have you tried a curl command verifying connectivity to the cluster with the parameters you're passing?

5

I am running these queries in cloudera and using curl command i just checked the connectivity to that node.. and yes it is pinging .
one more question - In which path should i load jar file ?

6

Not sure if you are using local disk of HDFS. But I just did a brew install of hive and then just added the JAR dependency locally like:

add jar file:///Users/wtaylor/Downloads/elasticsearch-hadoop-6.3.0.jar

7

Only in this path or should I place the jar file in all. The nodes of cloudera
My ES is installed in cloudera

8

JAR file should be accessible on the node or a path that Hive can access.

9

I have added in hive class path but still my cloudera is not accessing jar file.
More over i am still unable to find why my select query is not working?

10

Well if you can't access JAR your query won't work.

11

I am adding jar file using,
Add Jar hdfs:///user/impu/es/elasticsearch-hadoop-6.3.0.jar
command but i have to add jar for every session, so i want to add in class path.