Not able to copy SSL cert of elastic search certs using kibana docker container using docker-compose

Raushan · March 26, 2020, 9:20am

Hi team,

I need to create a Kibana docker container to work with elastic search with SSL ( Running on non-container env.) I am new on containerizing the application. Below is my docker-compose.yml file.

version: '3'
services:
  kibana:
    image: docker.elastic.co/kibana/kibana:7.5.2
    container_name: kibana
    environment:
      - SERVER_NAME=kibana
      - ELASTICSEARCH_HOSTS=https://app-espoc2-ssl.org.com:9200
      - XPACK_SECURITY_ENABLED=true
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD="XXXXXX"
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=["/data/DigiCert_Global_Root_CA.pem", /data/DigiCert_SHA2_Secure_Server_CA.pem]
      - ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=/data/orgPrivateKey.pem
      - SERVER_SSL_CERTIFICATE=/data/_.org.com.pem
    ports:
      - 5601:5601
    volumes:
      - ./ELK-certs/_.org.com.pem:/data/_.org.com.pem
      - ./ELK-certs/orgPrivateKey.pem:/data/orgPrivateKey.pem
      - ./ELK-certs/DigiCert_Global_Root_CA.pem:/data/DigiCert_Global_Root_CA.pem
      - ./ELK-certs/DigiCert_SHA2_Secure_Server_CA.pem:/data/DigiCert_SHA2_Secure_Server_CA.pem

When i run this i get a error:

ERROR: for kibana  Cannot start service kibana: error while creating mount source path '/host_mnt/c/work/APP2020/kibana/ELK-certs/DigiCert_Global_Root_CA.pem': mkdir /host_mnt/c/work/APP2020/kibana/ELK-certs/DigiCert_Global_Root_CA.pem: file exists

ERROR: for kibana  Cannot start service kibana: error while creating mount source path '/host_mnt/c/work/APP2020/kibana/ELK-certs/DigiCert_Global_Root_CA.pem': mkdir /host_mnt/c/work/APP2020/kibana/ELK-certs/DigiCert_Global_Root_CA.pem: file exists

Thanks in advance !!

Raushan · March 26, 2020, 9:46am

Please help. I have invested a lot of effort on this.

flash1293 · March 26, 2020, 10:13am

This problem doesn't seem to be related to Kibana, but to docker on windows. Maybe this SO thread can help you with your issue: https://stackoverflow.com/questions/50817985/docker-tries-to-mkdir-the-folder-that-i-mount

Raushan · March 26, 2020, 11:20am

Thanks @flash1293. Sharding D drive from my local has helped this. But when i run docker-compose up, i get the error that it's not copied to docker container.
Below is the error.

Attaching to kibana
kibana    | {"type":"log","@timestamp":"2020-03-26T11:13:58Z","tags":["info","plugins-system"],"pid":9,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,newsfeed,inspector,embeddable,advancedUiActions,expressions,eui_utils,data]"}
kibana    | {"type":"log","@timestamp":"2020-03-26T11:13:58Z","tags":["info","plugins","security"],"pid":9,"message":"Setting up plugin"}
kibana    | {"type":"log","@timestamp":"2020-03-26T11:13:58Z","tags":["warning","plugins","security","config"],"pid":9,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
kibana    | {"type":"log","@timestamp":"2020-03-26T11:13:58Z","tags":["fatal","root"],"pid":9,"message":"{ Error: ENOENT: no such file or directory, open '[\"/data/DigiCert_Global_Root_CA.pem\",'\n    at Object.openSync (fs.js:439:3)\n    at readFileSync (fs.js:344:35)\n    at readFile (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_client_config.js:114:49)\n    at Array.map (<anonymous>)\n    at parseElasticsearchClientConfig (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_client_config.js:117:63)\n    at new ClusterClient (/usr/share/kibana/src/core/server/elasticsearch/cluster_client.js:117:109)\n    at ElasticsearchService.createClusterClient (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:85:12)\n    at Object.createClient (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:68:21)\n    at Plugin.setup (/usr/share/kibana/x-pack/plugins/security/server/plugin.js:46:45)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '[\"/data/DigiCert_Global_Root_CA.pem\",' }"}
kibana    |
kibana    |  FATAL  Error: ENOENT: no such file or directory, open '["/data/DigiCert_Global_Root_CA.pem",'
kibana    |
kibana exited with code 1

flash1293 · March 26, 2020, 12:34pm

Seems like you are running into this problem: https://github.com/elastic/kibana/issues/39595:

You can't pass in values for arrays through environment variables.

You can try this workaround: https://github.com/elastic/kibana/issues/19773#issuecomment-395899975 (explicitly defining the array index for the individual values of the array) or putting the kibana configuration into a kibana.yml file on the host system and mount it into the right place in the container.

Raushan · March 26, 2020, 1:04pm

Okay...Thanks @flash1293 . Just got to know about that docker compose doesn't work in Kubernetes. So I need to drop with docker compose and start with Dockerfile with these set up of SSL.
Is the any post/article for this scenario ?

Please bear with me for all these question!!

flash1293 · March 26, 2020, 2:05pm

That actually makes things much easier - if you have a docker file you can use the docker.elastic.co/kibana/kibana:7.5.2 as your base and ADD your custom kibana.yml containing all these keys in standard yml (it should live at /usr/share/kibana/config/kibana.yml to get picked up correctly)

Then you won't have problems with the config array syntax

Raushan · March 27, 2020, 6:20am

Hi @flash1293,

Many thanks for helping.

I ran into another problem. After adding kibana.yml and starting the docker , it's not able to recognize path server.ssl.certificate: C:/ELK-certs/filename.com.pem . All other paths are read correctly.
Error is

{"type":"log","@timestamp":"2020-03-26T19:53:46Z","tags":["fatal","root"],"pid":6,"message":"{ Error: ENOENT: no such file or directory, open 'C:/ELK-certs/filename.com.pem'\n    at Object.openSync (fs.js:439:3)\n    at readFileSync (fs.js:344:35)\n    at getServerOptions (/usr/share/kibana/src/core/server/http/http_tools.js:77:34)\n    at HttpServer.setup (/usr/share/kibana/src/core/server/http/http_server.js:72:60)\n    at HttpService.runNotReadyServer (/usr/share/kibana/src/core/server/http/http_service.js:140:26)\n    at HttpService.setup (/usr/share/kibana/src/core/server/http/http_service.js:60:18)\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: 'C:/ELK-certs/filename.com.pem' }"}

server.ssl.certificate: C:/ELK-certs/filename.com.pem is creating problem.

Please find kibana.yml for the reference. https://pastebin.com/BJx5RHXg

Update : Kibana.yml : More readable

server.host: app-vm2-ssl.company.com
server.name: kinaba
elasticsearch.hosts: ["https://app-vm2-ssl.company.com:9200","https://app-vm1-ssl.company.com:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "111111"
server.ssl.enabled: true
server.ssl.key: C:/ELK-certs/companyPrivateKey.pem
server.ssl.certificate: C:/ELK-certs/filename.com.pem
elasticsearch.ssl.certificateAuthorities: ["C:/ELK-certs/DigiCert Global Root CA.pem", "C:/ELK-certs/DigiCert SHA2 Secure Server CA.pem"]
elasticsearch.ssl.verificationMode: full

Also, Is there any set up required in docker for ES server connectivity ?

flash1293 · March 27, 2020, 12:02pm

As Kibana will run within the containers, it won’t know about your C drive - you have to add the certificates to the image as well (eg under /certs/mycert) and then use that path in kibana.yml

system · April 24, 2020, 12:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.