SSL error when writing output of mapeduce to elasticsearch

Elastic Stack Elasticsearch
1

I am getting below mentioned error when running Mapreduce with output written to Elasticsearch. Please let me know what I am doing wrong.

Elasticsearch version: 2.0.0
elasticsearch-hadoop-mr: 2.2.0-beta1
Shield & marvel plugins installed

Steps to setup keystore on elasticsearch node:
1.keytool -importcert -keystore -file cacert.pem -alias snapin
2.keytool -genkey -alias node01 -keystore -keyalg RSA -keysize 2048 -validity 365
3.keytool -certreq -alias node01 -keystore -file -keyalg rsa
4.keytool -importcert -keystore -file -alias node01

Steps to setup keystore on client node:
1.keytool -genkey -alias client01 -keystore -keyalg RSA -keysize 2048 -validity 365
2.keytool -certreq -alias client01 -keystore -file -keyalg rsa
3.keytool -importcert -keystore -file -alias client01

SSL configuration parameters setup in elasticsearch.yml -
shield.ssl.keystore.path: /etc/elasticsearch/shield/
shield.ssl.keystore.password: *******
shield.transport.ssl: true
shield.http.ssl: true

Client side configuration:
Here are the client side configuration parameters that have been set-
es.nodes
es.resource
es.net.http.auth.user
es.net.http.auth.pass
es.net.ssl
es.net.ssl.keystore.location
es.net.ssl.keystore.pass

Error we are receiving:
2015-11-13 13:40:39,477 WARN mapred.Child [main]: Error running child
org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:363)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:367)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:121)
at org.elasticsearch.hadoop.rest.RestClient.esVersion(RestClient.java:513)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:177)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:378)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:591)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at com.aexp.imdc.elasticsearch.ContactImportDriver$ContactImportMapper.map(ContactImportDriver.java:70)
at com.aexp.imdc.elasticsearch.ContactImportDriver$ContactImportMapper.map(ContactImportDriver.java:32)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:702)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
at org.apache.hadoop.mapred.Child$4.run(Child.java:278)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at org.apache.hadoop.mapred.Child.main(Child.java:267)
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
at java.security.cert.PKIXParameters.(PKIXParameters.java:120)
at java.security.cert.PKIXBuilderParameters.(PKIXBuilderParameters.java:104)
at sun.security.validator.PKIXValidator.(PKIXValidator.java:88)
... 46 more
2015-11-13 13:40:39,485 INFO mapred.Task [main]: Runnning cleanup for the task

2

Looks like certificate problem on where the runtimes nodes where MapReduce runs.
This typically occurs if the certificates/trust store cannot be found or open correctly or if the JDK ca-certificates are not properly installed (on Linux issuing an sudo update-ca-certificates should be enough).

Do note that the trustStore need to be available on all nodes where the MapReduce code is running not just on the client where the job is started.

3

Thanks, Costin. We finally figured the issue. Looks like Elasticsearch is looking for the Java default truststore even though we have provided the truststore location and password. Once we imported the CA certificate into the Java default truststore, we do not see issue anymore. Do you know if this is a bug?

4

Hmm - that seems like a bug. Is it Elastic or ES-Hadoop? If you specify a truststore that one should be used without having to configure the default truststore (that's the whole point of using a dedicated truststore after all).

@srikrishna.alla Can you please post the exact versions of ES and Shield used?

@jaymode Do you why this might occur?

5

Can you share how you configured elasticsearch to use a truststore? What you are describing should not happen, so I would like to investigate how it could possibly happen.

6

Hi,

I am facing the same error while inserting into a hive table for which Storage is Elasticsearch.

Below are details of version:

Hadoop: Hortonworks 2.5.5
Hive: 1.2.1
ElasticeSearch: 5.2.0
Kibana: 5.2.1

Below are steps that I have taken:

--Add Elasticsearch in hive session.
ADD JAR elasticsearch-hadoop-5.2.0.jar; --THIS STEP IS SUCCESSFUL

CREATE EXTERNAL TABLE employeeDetail --THIS STEP IS SUCCESSFUL
(ID String,
Name String,
ZipCode String,
PhoneNumber String,
StartDate String)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'brij/Employee',
'es.nodes'='NODE_NAME',
'es.index.auto.create' = 'true',
'es.port'='443','es.net.ssl'='true','es.net.ssl.truststore.location'='/tmp/esdev.truststore',
'es.net.http.auth.user'='bsr7890','es.net.http.auth.pass'='******');

insert into table employeeDetail select * from someOtherTableWithSameStructure;

--Above mentioned Insert statement is running, but failing on the last reducer with below error

            VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
    --------------------------------------------------------------------------------
    Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
    Map 5 ..........   SUCCEEDED      1          1        0        0       0       0
    Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0
    Reducer 3 ......   SUCCEEDED      1          1        0        0       0       0
    Reducer 4             FAILED      1          0        0        1       4       0
    --------------------------------------------------------------------------------
    VERTICES: 04/05  [====================>>------] 80%   ELAPSED TIME: 17.35 s
    --------------------------------------------------------------------------------
    Status: Failed

Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Can you please help.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.