Kibana/enterprise search SSL connection error

[RRdev]

I'm trying to stablish an https connection between Kibana and Enterprise Search, but I'm having this error:

"Could not perform access check to Enterprise Search: FetchError: request to api/ent/v2/internal/client_config failed, reason: unable to get local issuer certificate".

I've found some similar issues here, but could not find a workable solution for my case

With ssl disabled, it works fine, but I really need ssl enabled. Here is part of the docker-compose configs.

Stack version: 8.5.0

elasticsearch:
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=elasticsearch
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/privkey.pem
      - xpack.security.http.ssl.certificate=certs/fullchain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/privkey.pem
      - xpack.security.transport.ssl.certificate=certs/fullchain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}

  kibana:
    container_name: kibana
    environment:
      - SERVERNAME=myhost
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=/usr/share/elasticsearch/config/certs/fullchain.pem
      - SERVER_SSL_KEY=/usr/share/elasticsearch/config/certs/privkey.pem
      - SERVER_PUBLICBASEURL=https://myhost:5601
      - ELASTICSEARCH_HOSTS=https://myhost:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_ENABLED=true
      - ENTERPRISESEARCH_HOST=https://myip:3002
      - XPACK_SECURITY_ENCRYPTIONKEY=*************
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=*************
      - XPACK_REPORTING_ENCRYPTIONKEY=*************
     - XPACK_REPORTING_KIBANASERVER_HOSTNAME=myhost
    
enterprisesearch:
    environment:
      - SERVERNAME=myhost
      - secret_management.encryption_keys=[encryption_keys]
      - allow_es_settings_modification=true
      - elasticsearch.host=https://myhost:9200
      - elasticsearch.username=elastic
      - elasticsearch.password=${ELASTIC_PASSWORD}
      - elasticsearch.ssl.enabled=true
      - kibana.external_url=https://myhost:5601
      - kibana.host=https://myhost:5601
      - kibana.startup_retry.enabled=true
      - kibana.startup_retry.interval=5
      - kibana.startup_retry.fail_after=60
      - ent_search.ssl.enabled=true
      - ent_search.external_url=https://myip:
     - ent_search.ssl.keystore.path=/usr/share/enterprise-search/cacert/keystore.jks
      - ent_search.ssl.keystore.password=KEYSTORE_PASSWORD
      - ent_search.ssl.keystore.key_password=KEYSTORE_PASSWORD
      - ent_search.listen_port=3002
          
volumes:
  enterprisesearchdata:
  esdata:
  kibanadata:

[Rich_Kuzsma]

Hello RRdev, thanks for your question. Sorry you're running into difficulty configuring this, it isn't easy and I'm sure you're not the only one struggling.

When configuring TLS/SSL between Enterprise Search and Kibana, we have to consider TLS/SSL traffic going in both directions: from the Enterprise Search server to the Kibana server, and from the Kibana server to the Enterprise Search server.

When the Enterprise Search server connects to the Kibana server via the kibana.host URL specified in config/enterprise-search.yml, Enterprise Search connects using the Elasticsearch TLS/SSL settings configured in config/enterprise-search.yml:

elasticsearch.ssl.enabled: 
elasticsearch.ssl.certificate:
elasticsearch.ssl.certificate_authority:
elasticsearch.ssl.key:
elasticsearch.ssl.key_passphrase:
elasticsearch.ssl.verify: true

There isn't currently a way to configure different TLS/SSL settings for Enterprise Search to use when connecting to Kibana. Enterprise Search always uses the Elasticsearch TLS/SSL settings.

Note that the Enterprise Search server only connects to the Kibana server for the purposes of checking telemetry settings and verify the version of Kibana.

Going in the other direction... When the Kibana server connects to the Enterprise Search server over TLS/SSL, Kibana must be configured to trust Enterprise Search's certificate authority. There is documentation explaining how to configure the Enterprise Search TLS/SSL certs in Kibana. Specifically, these settings need to be configured in Kibana config/kibana.yml:

enterpriseSearch.host: https://some-host.tld:3002
enterpriseSearch.ssl.verificationMode: certificate
enterpriseSearch.ssl.certificateAuthorities:
  - /path/to/your/ca.pem

When running in docker, you can specify most Kibana configuration settings using environment variables. For example, specify enterpriseSearch.host as the ENTERPRISESEARCH_HOST env var (replace . with _ and use uppercase).

Unfortunately, the enterpriseSearch.ssl.verificationMode and enterpriseSearch.ssl.certificateAuthorities are not listed in the Kibana docker ENV variable settings mapping list.

I also don't see these variables documented in the Kibana documentation for supported Enterprise Search configuration settings.

I'll file an issue to fix the docs and add these settings to the ENV var mappings list in a future release.

In the interim, it may be possible to specify an entire kibana.yml file with all the necessary settings for Docker using a bind-mounted configuration file.

Hope this helps,
Rich

[RRdev]

Hi Rich_Kuzsma, thank you very much for your reply, I appreciate it.
I'll try your suggestions and I'll keep up with documentation updates.

Thanks!