Experts,
I am trying to push data from a Hive table over to ElasticSearch. There are numerous examples on the Internet of how to do this, but there are very few on how to do it when security is enabled on ElasticSearch. I have tried numerous combinations of properties for the TBL_PROPERTIES portion of the "CREATE EXTERNAL TABLE" but nothing works.
Here is my current attempt:
create external table sample_07_es (code string, description string, total_emp int, salary int) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'job/jobs','es.nodes'='mapr04.wired.carnoustie','es.index.auto.create' = 'true','es.net.ssl'='true','es.net.ssl.truststore.location'='/opt/mapr/elasticsearch/elasticsearch-6.2.3/etc/elasticsearch/keystores','es.net.http.auth.user'='admin','es.net.http.auth.pass'='admin');
And here is the error message in the Hive logs:
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Expected to find keystore file at [/opt/mapr/elasticsearch/elasticsearch-6.2.3/etc/elasticsearch/keystores] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.
at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:173)
at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:158)
at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:127)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:478)
So, the error is around keystores, which are not my strongest area. I don't know what keystore Hive is looking for, but the error is coming from the ElasticSearch method createSSLContext, so my guess was that I needed to pass the ElastiicSearch keystores.
If I knew what keystore it was looking for, I could better troubleshoot the issue. I don't know if the keystore path should be a path on the local machine or a keystore path in the MapR FS.
My next step is to start reading through the code, I "think" this is the code that is being called:
KeyStore truststore = KeyStore.getInstance("jks");
try (InputStream is = Files.newInputStream(keyStorePath)) {
truststore.load(is, keyStorePass.toCharArray());
}
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadTrustMaterial(truststore, null);
final SSLContext sslContext = sslBuilder.build();
RestClientBuilder builder = RestClient.builder(
new HttpHost("localhost", 9200, "https"))
.setHttpClientConfigCallback(new HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setSSLContext(sslContext);
}
});
Welcome any thoughts or ideas!