Issue with pki elastic search hadoop


(Srikrishna Alla) #1

Hi,

I am having an issue accessing hive table when pki is turned on. These are the steps I have followed for setting up pki-

  1. Created a CA
    a. Generate a self-signed CA certificate to establish your CA as an authority. (Make sure to remember the PEM passphrase) openssl req -new -x509 -extensions v3_ca -keyout private/cakey.pem -out certs/cacert.pem -days 1460 -config conf/caconfig.cnf
  2. Copied the cacert.pem to all data nodes.
  3. Import our CA cert into a keystore on each elastic search node
    a. keytool -importcert -keystore /etc/elasticsearch/shield/.jks -file /etc/elasticsearch/shield/cacert.pem -alias
  4. Generate a private key for our elastic search nodes and generate a csr for our elastic search nodes.NOTE Make sure to include the common name for each csr and make it unique(i.e. short host name), this is needed for signing as each csr needs to be unique.)END NOTE
    a. keytool -genkey -alias XXXX -keystore es-hadoop.jks -keyalg RSA -keysize 2048 -validity 365 -ext san=dns:xxxx,dns:xxx.xxx.com,ip:10.xx.xx.xx
  5. Copy all csrs to the CA system for signing:
  6. Sign all of the csr’s.
    a. openssl ca -in .csr -notext -out -signed.crt -config conf/caconfig.cnf -extensions v3_req
  7. send all of the signed certificates back to the original hosts.
  8. Import the signed certs into the keystore.
    a. keytool -importcert -keystore /opt/mapr/hadoop/es-hadoop.jks -file XXXX.csr-signed.crt -alias XXXX

I am using elasticsearch-hadoop-2.2.0-beta1.jar & table properties as -
ROW FORMAT SERDE
'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY
'org.elasticsearch.hadoop.hive.EsStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
TBLPROPERTIES
(
'es.nodes'='xxxx.xxxx.com:9200',
'es.resource'='xxx/xxxxx',
'es.net.ssl'='true',
'es.net.ssl.keystore.location'='file:///es_hadoop.jks',
'es.net.ssl.keystore.pass'=''
);

Error we are getting is:
Failed with exception java.io.IOException:org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: missing authentication token for REST request [/]


(Costin Leau) #2

I recommend breaking the process into two steps:

  1. configure shield. Use cUrl or other tools to check that http is enabled followed by checking the PKI setup.
  2. configure ES-Hadoop. Do note that you should specify your keystore in HDFS or other globally available location on all nodes (file:/// should be used only if all nodes on which the task runs have the file).

(Srikrishna Alla) #3

Thanks, Costin. Will try it out.

Thanks,
Sri


(system) #4