HI Team,
Is there a way we can use the ES-Hadoop connector to query using https.
I tried to query the elastic search using the below configurations
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_HOST, var_elastic_ip)
sparkConf.set(ConfigurationOptions.ES_PORT, var_elastic_port)
sparkConf.set(ConfigurationOptions.ES_RESOURCE, var_ES_Index_Type)
sparkConf.set(ConfigurationOptions.ES_QUERY, var_ES_Query_String)
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_USER, var_ES_user)
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_PASS, var_ES_pass)
sparkConf.set(ConfigurationOptions.ES_NET_SSL_PROTOCOL, "SSL")
:
:
:
val var_Read_Data_From_ES_RDD = EsSpark.esJsonRDD(sc)
but it is throwing the below error:
16/11/16 16:59:49 DEBUG HttpMethodBase: Default charset used: UTF-8
16/11/16 16:59:49 TRACE CommonsHttpTransport: Rx @[X.X.X.X] [400-Bad Request] [
400 Bad Request
The plain HTTP request was sent to HTTPS portnginx/1.4.6 (Ubuntu) ] 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 TRACE CommonsHttpTransport: Closing HTTP transport to X.X.X.X:9200 16/11/16 16:59:49 TRACE HttpConnection: enter HttpConnection.close() 16/11/16 16:59:49 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams() Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:190) at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:231)