Hi,
I am trying to configure a system to use both Basic Authentication and
HTTPS to Store data to ElasticSearch.
My system is configured with a Pig script running through Hadoop to connect
to Apache (configured as a proxy) to forward the request to ElasticSearch.
Using simple HTTP and Basic Authentication works correctly. However, when I
try to force my ES UDF to use HTTPS, I get errors in my Apache logs and my
job fails.
The relevant snippet of my Pig script is below:
REGISTER
/bigdata/cloudera/ES_HadoopJar/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar
DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage(
'es.nodes=https://127.0.0.1:28443',
'es.net.proxy.http.host=https://127.0.0.1',
'es.net.proxy.http.port=28443',
'es.net.proxy.http.user=myuser',
'es.net.proxy.http.pass=mypass',
'es.http.retries=10');
data = LOAD... ...
STORE data INTO 'my_data_index/data' USING EsStorage;
The error output to the Apache log is as follows:
SSL Library Error: error:1407609B:SSL
routines:SSL23_GET_CLIENT_HELLO:https proxy request -- speaking HTTP to
HTTPS port!?
The error/stacktrace from my Map job is as follows:
Error: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException:
Connection error (check network and/or proxy settings)- all nodes failed;
tried [[https://127.0.0.1:28443]] at
*org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:123)
at *
*org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:300) at *
*org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:284) at *
*org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:288) at *
*org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:117) at *
*org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:99)
at *
*org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:59)
at *
*org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:180)
at *
*org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:157)
at *
*org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:196) at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
at *
*org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635)
at *
*org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at *
*org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at *
*org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at *
*org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at *
*org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at *
*org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at *
*org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at *
*java.security.AccessController.doPrivileged(Native Method) at *
*javax.security.auth.Subject.doAs(Subject.java:415) at *
*org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at *
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
So my question is, is this possible (i.e. can it work)? And if so, where am
I going wrong?
Thanks in advance for any help.
Aidan
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d19a21e-2947-4ac4-8e3e-68cc8c25185b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.