Hadoop connection with Elasticsearch

Hi,

While trying to connect to elasticsearch through hive, getting below exception

ERROR : Job Submission failed with exception 
'org.elasticsearch.hadoop.EsHadoopIllegalArgumentException(Cannot detect ES 
version - typically this happens if the network/Elasticsearch cluster is not 
accessible or when targeting a WAN/Cloud instance without the proper setting 
'es.nodes.wan.only')'  
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES 
version - typically this happens if the network/Elasticsearch cluster is not 
accessible or when targeting a WAN/Cloud instance without the proper setting 
'es.nodes.wan.only'
    at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:344)
    at org.elasticsearch.hadoop.mr.EsOutputFormat.init(EsOutputFormat.java:262)
    at org.elasticsearch.hadoop.mr.EsOutputFormat.checkOutputSpecs(EsOutputFormat.java:253)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:1087)
    at org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:272)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:428)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
    at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
    at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: 
Connection error (check network and/or proxy settings)- all nodes failed; tried 
[[127.0.0.1:9200]] 
    at 
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:152)
    at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:403)
    at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:367)
    at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:371)
    at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:166)
    at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:692)
    at 

org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:334)
... 40 more

ERROR : FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
INFO  : Completed executing 
command(queryId=hive_20191010211212_5c6fbe43-b34e-49be-8ea3- 

42ff5f305978); Time taken: 0.103 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

Using hadoop-elasticsearch connector version - 6.7.0
Elasticsearch - 6.7.0

Also I dont understand why its trying to connect to 127.0.0.1 when i created external table on
CREATE EXTERNAL TABLE test.artist (name STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.node' = '10.34.9.64' , 'es.resource' = 'test/logs',
'es.index.auto.create' = 'yes');

Need some urgent help. Thanks

The property you want is es.nodes not es.node - hopefully that helps!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.