EShadoopconnector connection to HTTPS instead of HTTP


(Praveen) #1

HI Team,

Is there a way we can use the ES-Hadoop connector to query using https.

I tried to query the elastic search using the below configurations
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_HOST, var_elastic_ip)
sparkConf.set(ConfigurationOptions.ES_PORT, var_elastic_port)
sparkConf.set(ConfigurationOptions.ES_RESOURCE, var_ES_Index_Type)
sparkConf.set(ConfigurationOptions.ES_QUERY, var_ES_Query_String)
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_USER, var_ES_user)
sparkConf.set(ConfigurationOptions.ES_NET_PROXY_HTTPS_PASS, var_ES_pass)
sparkConf.set(ConfigurationOptions.ES_NET_SSL_PROTOCOL, "SSL")
:
:
:
val var_Read_Data_From_ES_RDD = EsSpark.esJsonRDD(sc)

but it is throwing the below error:
16/11/16 16:59:49 DEBUG HttpMethodBase: Default charset used: UTF-8
16/11/16 16:59:49 TRACE CommonsHttpTransport: Rx @[X.X.X.X] [400-Bad Request] [

400 The plain HTTP request was sent to HTTPS port

400 Bad Request

The plain HTTP request was sent to HTTPS port
nginx/1.4.6 (Ubuntu) ] 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 DEBUG HttpMethodBase: re-creating response stream from byte array 16/11/16 16:59:49 TRACE CommonsHttpTransport: Closing HTTP transport to X.X.X.X:9200 16/11/16 16:59:49 TRACE HttpConnection: enter HttpConnection.close() 16/11/16 16:59:49 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams() Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:190) at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:231)

(James Baiera) #2

Is es.net.ssl enabled currently? If that option is not enabled the connector ignores any SSL settings and communicates without SSL.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.