Elasticsearch SSL certificate error - javax.crypto.BadPaddingException: Given final block not properly padded

[fgomes (Filipe Gomes)]

I'm trying to get user management and authentication in Kibana. For that it is required to enable the security module, and since we're running a 3 node cluster, SSL security must be enabled as well.

I tried to use Dockerfile to create my own elasticsearch image with built in ssl certificate file. But it outputs the following error message:

Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Dockerfile:

FROM docker.elastic.co/elasticsearch/elasticsearch:6.8.9

RUN bin/elasticsearch-certutil ca --out config/elastic-stack-ca.p12 --pass "password"

RUN bin/elasticsearch-certutil cert --ca-pass "password" config/elastic-stack-ca.p12 --out config/elastic-certificates.p12 --pass "password"

RUN chgrp -R 0 config/elastic-certificates.p12

RUN chmod +rw /usr/share/elasticsearch/config/elastic-certificates.p12

EXPOSE 9200

This is my service definition, here is as 2 nodes example that I've been using for proof of concept work:

elastic.yml:

version: '3.4'

services:
  es01:
    image: myelasticsearch:1.0
    environment:
      - node.name=es01
      - discovery.zen.minimum_master_nodes=2
      - elastisearch.password=password
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
    volumes:
      - esdata_01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    healthcheck:
      test: curl --cacert /usr/share/elasticsearch/config/elastic-stack-ca.p12 -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  es02:
    image: myelasticsearch:1.0
    environment:
      - node.name=es02
      - discovery.zen.minimum_master_nodes=2
      - elastisearch.password=password
      - discovery.zen.ping.unicast.hosts=es01
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
    volumes:
      - esdata_02:/usr/share/elasticsearch/data

volumes:
  esdata_01:
  esdata_02:

After reading the documentation, I can't see what I'm doing wrong.

[TimV (Tim Vernum)]

@fgomes

--pass "password"

I don't see anywhere where you configure Elasticsearch with this password.
You need to set xpack.security.transport.ssl.keystore.secure_password and xpack.security.transport.ssl.truststore.secure_password in the ES keystore.

[fgomes (Filipe Gomes)]

Thanks for your help. The ES nodes are now with security and ssl enabled. This is what I've ended up defining:

Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:6.8.9

RUN bin/elasticsearch-certutil ca --out config/elastic-stack-ca.p12 --pass "password"

RUN bin/elasticsearch-certutil cert --ca-pass "password" config/elastic-stack-ca.p12 --out config/elastic-certificates.p12 --pass "password"

RUN chgrp -R 0 config/elastic-certificates.p12

RUN chmod +rw /usr/share/elasticsearch/config/elastic-certificates.p12

EXPOSE 9200

Stack File:

version: '3.4'

services:
  es01:
    hostname: es01
    image: myelasticsearch:1.0
    environment:
      - node.name=es01
      - discovery.zen.minimum_master_nodes=2
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - ELASTIC_PASSWORD=password
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.http.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.http.ssl.keystore.password=password
      - xpack.security.http.ssl.truststore.password=password
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.keystore.password=password
      - xpack.security.transport.ssl.truststore.password=password
    volumes:
      - esdata_01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    healthcheck:
      test: curl --cacert /usr/share/elasticsearch/config/elastic-stack-ca.p12 -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  es02:
    hostname: es02
    image: myelasticsearch:1.0
    environment:
      - node.name=es02
      - discovery.zen.minimum_master_nodes=2
      - discovery.zen.ping.unicast.hosts=es01
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - ELASTIC_PASSWORD=password
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.http.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.http.ssl.keystore.password=password
      - xpack.security.http.ssl.truststore.password=password
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/elastic-certificates.p12
      - xpack.security.transport.ssl.keystore.password=password
      - xpack.security.transport.ssl.truststore.password=password
    volumes:
      - esdata_02:/usr/share/elasticsearch/data

volumes:
  esdata_01:
  esdata_02:

Next task is configuring Kibana to use the same certificate.