Security error in kibana container

[jrcartmell (JohnCartmell)]

Below is an error I am getting in my kibana container. And at the bottom is my yaml config file. I am trying to run 3 instances of ES and 1 Kibana with security enabled.

Error:

{"type":"log","@timestamp":"2022-04-19T16:02:13Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2022-04-19T16:02:13Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2022-04-19T16:02:14Z","tags":["warning","config","deprecation"],"pid":7,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2022-04-19T16:02:14Z","tags":["fatal","root"],"pid":7,"message":"Error: error:0909006C:PEM routines:get_name:no start line\n    at Object.createSecureContext (_tls_common.js:156:17)\n    at Server (_tls_wrap.js:903:27)\n    at new Server (https.js:62:14)\n    at Object.createServer (https.js:85:10)\n    at module.exports.internals.Core._createListener (/usr/share/kibana/node_modules/hapi/lib/core.js:491:79)\n    at new module.exports.internals.Core (/usr/share/kibana/node_modules/hapi/lib/core.js:112:30)\n    at new module.exports (/usr/share/kibana/node_modules/hapi/lib/server.js:25:18)\n    at createServer (/usr/share/kibana/src/core/server/http/http_tools.js:113:18)\n    at HttpServer.setup (/usr/share/kibana/src/core/server/http/http_server.js:86:48)\n    at HttpService.runNotReadyServer (/usr/share/kibana/src/core/server/http/http_service.js:162:26)\n    at HttpService.setup (/usr/share/kibana/src/core/server/http/http_service.js:78:18)"}

 FATAL  Error: error:0909006C:PEM routines:get_name:no start line

Yaml config file:

version: '3.7'

networks:
  default:
    driver: bridge
  john:
    driver: bridge

volumes:
  johnsearch-data01:
    driver: local
  johnsearch-data02:
    driver: local
  certs:
    external: true

services:
  elasticsearch_certificates:
    container_name: elasticsearch_certificates
    image:  docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    environment:
      CA_PASSWORD: $CA_PASSWORD
      ELASTICSEARCH01_CERT_PASSWORD: $ELASTICSEARCH01_CERT_PASSWORD
      ELASTICSEARCH02_CERT_PASSWORD: $ELASTICSEARCH02_CERT_PASSWORD
      ELASTICSEARCH03_CERT_PASSWORD: $ELASTICSEARCH03_CERT_PASSWORD
      KIBANA_CERT_PASSWORD: $KIBANA_CERT_PASSWORD
    command: >
      bash -c '
        if [[ ! -f /certs/elasticsearch01.p12 || \
              ! -f /certs/elasticsearch02.p12 || \
              ! -f /certs/elasticsearch03.p12 || \
              ! -f /certs/kibana.zip ]]; then
          echo "Removing certificates" &&
          rm -rf /certs/* &&

          echo "Generating CA" &&
          bin/elasticsearch-certutil ca --silent --pass ${CA_PASSWORD} --pem --out /certs/ca.zip &&
          unzip /certs/ca.zip -d /certs &&

          echo "Generating certificate for Elasticsearch01" &&
          bin/elasticsearch-certutil cert --silent --ca-cert /certs/ca/ca.crt --ca-key /certs/ca/ca.key --ca-pass ${CA_PASSWORD} --pass ${ELASTICSEARCH01_CERT_PASSWORD} --dns elasticsearch01 --out /certs/elasticsearch01.p12 &&

          echo "Generating certificate for Elasticsearch02" &&
          bin/elasticsearch-certutil cert --silent --ca-cert /certs/ca/ca.crt --ca-key /certs/ca/ca.key --ca-pass ${CA_PASSWORD} --pass ${ELASTICSEARCH02_CERT_PASSWORD} --dns elasticsearch02 --out /certs/elasticsearch02.p12 &&

          echo "Generating certificate for Elasticsearch03" &&
          bin/elasticsearch-certutil cert --silent --ca-cert /certs/ca/ca.crt --ca-key /certs/ca/ca.key --ca-pass ${CA_PASSWORD} --pass ${ELASTICSEARCH02_CERT_PASSWORD} --dns elasticsearch03 --out /certs/elasticsearch03.p12 &&

          echo "Generating certificate for Kibana" &&
          bin/elasticsearch-certutil cert --silent --ca-cert /certs/ca/ca.crt --ca-key /certs/ca/ca.key --ca-pass ${CA_PASSWORD} --pass ${KIBANA_CERT_PASSWORD} --pem --dns kibana --out /certs/kibana.zip &&
          unzip /certs/kibana.zip -d /certs &&
          mv /certs/instance/instance.crt /certs/kibana.crt &&
          mv /certs/instance/instance.key /certs/kibana.key &&
          rm -rf /certs/instance &&

          chown -R 1000:0 /certs &&

          sleep 5m

        fi;
      '
    user: "0"
    working_dir: /usr/share/elasticsearch
    volumes:
      - certs:/certs

  johnsearch01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: john_es01
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - node.name=john_es01
      - cluster.name=john_es-cluster
      - network.host=0.0.0.0
      - discovery.seed_hosts=john_es02,john_es03
      - cluster.initial_master_nodes=john_es01,john_es02,john_es03
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.authc.token.enabled=true
      - xpack.security.audit.enabled=true
      - xpack.security.authc.realms.file.file1.order=0
      - xpack.security.authc.realms.native.native1.order=1
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.keystore.password=$ELASTICSEARCH01_CERT_PASSWORD
      - xpack.security.transport.ssl.truststore.password=$ELASTICSEARCH01_CERT_PASSWORD
      - xpack.security.transport.ssl.keystore.path=certs/elasticsearch01.p12
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.password=$ELASTICSEARCH01_CERT_PASSWORD
      - xpack.security.http.ssl.truststore.password=$ELASTICSEARCH01_CERT_PASSWORD
      - xpack.security.http.ssl.keystore.path=certs/elasticsearch01.p12
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=trial
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /john_data/test/volumes/johnsearch-data01:/usr/share/johnsearch/data
      - certs:/usr/share/elasticsearch/config/certs/:ro
    ports:
      - 9210:9200
      - 9310:9300
    networks:
      - john
    healthcheck:
      test: [
        "CMD", "test",
        "-f", "config/certs/elasticsearch01.p12", "-a",
        "-f", "config/certs/elasticsearch02.p12", "-a",
        "-f", "config/certs/elasticsearch03.p12", "-a",
        "-f", "config/certs/kibana.crt", "-a",
        "-f", "config/certs/kibana.key"
      ]
      interval: 15s
      timeout: 10s
      retries: 10


  johnsearch02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: john_es02
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - node.name=john_es02
      - cluster.name=john_es-cluster
      - network.host=0.0.0.0
      - discovery.seed_hosts=john_es01,john_es03
      - cluster.initial_master_nodes=john_es01,john_es02,john_es03
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.authc.token.enabled=true
      - xpack.security.audit.enabled=true
      - xpack.security.authc.realms.file.file1.order=0
      - xpack.security.authc.realms.native.native1.order=1
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.keystore.password=$ELASTICSEARCH02_CERT_PASSWORD
      - xpack.security.transport.ssl.truststore.password=$ELASTICSEARCH02_CERT_PASSWORD
      - xpack.security.transport.ssl.keystore.path=certs/elasticsearch02.p12   # Change from 01
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.password=$ELASTICSEARCH02_CERT_PASSWORD
      - xpack.security.http.ssl.truststore.password=$ELASTICSEARCH02_CERT_PASSWORD
      - xpack.security.http.ssl.keystore.path=certs/elasticsearch02.p12  # Changed from 01
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=trial
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /john_data/test/volumes/johnsearch-data02:/usr/share/johnsearch/data
      - certs:/usr/share/elasticsearch/config/certs/:ro
    depends_on:
      - johnsearch01
    networks:
      - john


  johnsearch03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: john_es03
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - node.name=john_es03
      - cluster.name=john_es-cluster
      - network.host=0.0.0.0
      - discovery.seed_hosts=john_es01,john_es02
      - cluster.initial_master_nodes=john_es01,john_es02,john_es03
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.authc.token.enabled=true
      - xpack.security.audit.enabled=true
      - xpack.security.authc.realms.file.file1.order=0
      - xpack.security.authc.realms.native.native1.order=1
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.keystore.password=$ELASTICSEARCH03_CERT_PASSWORD
      - xpack.security.transport.ssl.truststore.password=$ELASTICSEARCH03_CERT_PASSWORD
      - xpack.security.transport.ssl.keystore.path=certs/elasticsearch03.p12   # Change from 01
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.password=$ELASTICSEARCH03_CERT_PASSWORD
      - xpack.security.http.ssl.truststore.password=$ELASTICSEARCH03_CERT_PASSWORD
      - xpack.security.http.ssl.keystore.path=certs/elasticsearch03.p12  # Changed from 01
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=trial
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /john_data/test/volumes/johnsearch-data03:/usr/share/johnsearch/data
      - certs:/usr/share/elasticsearch/config/certs/:ro
    depends_on:
      - johnsearch01
    networks:
      - john
 
  johnkibana:
    image: docker.elastic.co/kibana/kibana:7.10.1
    container_name: john_kibana

    environment:
      - ELASTICSEARCH_HOSTS=["https://john_es01:9200","https://john_es02:9200","https://john_es03:9200"]
      - XPACK_SECURITY_ENABLED=true
      - ELASTICSEARCH_KIBANA_SYSTEM=elastic
      - ELASTICSEARCH_PASSWORD=$ELASTIC_PASSWORD
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=config/certs/kibana.key
      - SERVER_SSL_CERTIFICATE=config/certs/kibana.crt
      - SERVER_SSL_PASSWORD=${KIBANA_CERT_PASSWORD}
    volumes:
      - certs:/usr/share/kibana/config/certs/:ro
    healthcheck:
      test: [
        "CMD", "test",
        "-f", "config/certs/elasticsearch01.p12", "-a",
        "-f", "config/certs/elasticsearch02.p12", "-a",
        "-f", "config/certs/elasticsearch03.p12", "-a",
        "-f", "config/certs/kibana.crt", "-a",
        "-f", "config/certs/kibana.key"
      ]
      interval: 15s
      timeout: 10s
      retries: 10
    ports:
      - 5611:5601
    depends_on:
      - johnsearch01
    networks:
      - john

I am a complete novice with ES and Kibana, so I am kind of lost as to what to do next or try. Any help would be appreciated. Thanks.

[ffknob (Flávio Knob)]

Take a look at this answer from StackOverflow, it might help:

Error: error:0909006C:PEM routines:get_name:no start line - node

One other thing, when you are generating the certificates you are passing parameters like --dns elasticsearch01, but I think your internal docker name resolution will create containers with hostnames like john_es01 (it will take it from the container_name of each service in your docker-compose file).

I think the problem here is Kibana is expecting your cert file to be in PEM format, but it is not. What is the content of the config/certs/kibana.crt file?

This is another thread where the user got the exactly the same error you are getting in Kibana:

How to create security certificate pem for elasticsearch an kibana

Also I am not a helm user or expert.