ES(ECE 2.2) - HADOOP Connectivity

rijash · August 8, 2019, 3:08pm

Hi,

I am getting the below error while trying to ingest data to ES (ECE) from hadoop (HIVE) query.

===============
2019-08-08 13:55:24,957 INFO [main]: httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(444)) - Retrying request

2019-08-08 13:55:24,958 ERROR [main]: rest.NetworkClient (NetworkClient.java:execute(147)) - Node [d985e2b66da74bd1860a09a9347c3506.elkeu.ondemand.com:443] failed (java.net.ConnectException: Connection refused (Connection refused)); no other nodes left - aborting...

2019-08-08 13:55:24,959 ERROR [main]: CliDriver (SessionState.java:printError(1089)) - Failed with exception java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

java.io.IOException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:520)

at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:427)

at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)

at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1773)

at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:237)

at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:233)

at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:340)

at org.elasticsearch.hadoop.hive.HiveUtils.init(HiveUtils.java:197)

at org.elasticsearch.hadoop.hive.EsHiveInputFormat.getSplits(EsHiveInputFormat.java:112)

at org.elasticsearch.hadoop.hive.EsHiveInputFormat.getSplits(EsHiveInputFormat.java:51)

at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:371)

at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:303)

at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:458)

... 15 more

Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[d985e2b66da74bd1860a09a9347c3506.elkeu.ondemand.com:443]]

at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:152)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:424)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)

at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)

at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:735)

at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)

... 21 more

2019-08-08 13:55:24,959 INFO [main]: exec.TableScanOperator (Operator.java:close(616)) - Closing operator TS[0]

2019-08-08 13:55:24,959 INFO [main]: exec.SelectOperator (Operator.java:close(616)) - Closing operator SEL[1]

2019-08-08 13:55:24,959 INFO [main]: exec.ListSinkOperator (Operator.java:close(616)) - Closing operator OP[3]

2019-08-08 13:55:24,963 INFO [main]: CliDriver (SessionState.java:printInfo(1066)) - Time taken: 0.051 seconds

2019-08-08 13:55:24,963 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) -

2019-08-08 13:55:24,963 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=releaseLocks start=1565272524963 end=1565272524963 duration=0 from=org.apache.hadoop.hive.ql.Driver>

Below configuration is working fine for ES as independent component but failing in ECE.

====================
'es.index.read.missing.as.empty'='true',
'es.mapping.date.rich'='false',
'es.mapping.names'='date:@timestamp',
'es.net.http.auth.pass'='<es_cluster_pwd>',
'es.net.http.auth.user'='<es_cluster_id>',
'es.net.ssl'='true',
'es.net.ssl.cert.allow.self.signed'='true',
'es.net.ssl.keystore.location'='file:///etc/ssl/certs/1.jks',
'es.net.ssl.keystore.pass'='<jks_pwd>',
'es.net.ssl.keystore.type'='jks',
'es.net.ssl.protocol'='SSL',
'es.nodes.wan.only'='true',
'es.nodes'='d985e2b66da74bd1860a09a9347c3506.elkeu.ondemand.com:443',
'es.query'='?q=*',
'es.resource'='users/user-events'

james.baiera · August 14, 2019, 7:20pm

Can you increase the logging level on Hive and Hadoop for just the org.elasticsearch.hadoop.rest.commonshttp package to TRACE and post the output here? There seems to be an issue with the initial request from ES-Hadoop to the ES cluster failing to complete.

rijash · August 14, 2019, 8:58pm

Hi James,

Thanks for your response. I have fixed the issue by configuring below in the table properties.

'es.index.read.missing.as.empty'='true',
'es.mapping.date.rich'='false',
'es.mapping.names'='date:@timestamp',
'es.net.http.auth.pass'='<ES_PWD>',
'es.net.http.auth.user'='<ES_USERNAME>',
'es.net.ssl'='true',
'es.net.ssl.cert.allow.self.signed'='true',
'es.net.ssl.keystore.location'='<JKS_FILE_PATH>',
'es.net.ssl.keystore.pass'='<KEYSTORE_PWD>',
'es.net.ssl.keystore.type'='jks',
'es.net.ssl.truststore.location'='<JKS_FILE_PATH>',
'es.net.ssl.truststore.pass'='<KEYSTORE_PWD>',
'es.net.ssl.truststore.type'='jks',
'es.net.ssl.protocol'='SSL',
'es.nodes.wan.only'='true',
'es.nodes'='<ES_CLUSTER_ID>:9243',
'es.query'='?q=*',
'es.resource'='<INDEX_NAME>/<DOCUMENT_TYPE>'

Regards,
Rijash

system · September 11, 2019, 8:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.