Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

Hello,

I generated with certutils my own ca.crt, ca.key and the certificates for each node(with an instances.yml), however when I load the elastic and kibana, kibana docker container logs says:

kibana_1 | [2024-09-17T12:15:58.476+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

In order to check if the ca.crt and credentials are correct from inside the Kibana container, I do in a shell(inside kibana container):

curl --cacert /usr/share/kibana/config/certs/ca/ca.crt -v https://es01:9200 -u kibana_system:xxxxx

and the elastic node response is correct, so I'm not sure why Kibana is not able to connect to elasticsearch. But I can say that the certificate authority is good. Just to note that I used

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system inside the elastic container in order to set the kibana password for that user and after that I modified the docker-compose.yml and finally restart the kibana container.

Can someone help me? thanks!

My docker-compose file:

# The version of the Docker Compose file format
version: "2.2"
# The services that make up the application
services:
  # The setup service is responsible for setting up the Elasticsearch cluster
  setup:
    # The Docker image to use for this service
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    # The volumes to mount into this service
    volumes:
      - ./config/certs:/usr/share/elasticsearch/config/certs
    # The user to run commands as inside the container
    user: "0"
    # The command to run inside the container
    #command: tail -f '/dev/null'
      
    # The healthcheck to determine the health of this service
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  # The es01 service represents an Elasticsearch node in the cluster
  es01:
    # This service depends on the setup service
    depends_on:
     setup:
       condition: service_healthy

    # The Docker image to use for this service
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    # The volumes to mount into this service
    volumes:
      - ./config/certs:/usr/share/elasticsearch/config/certs
      - esdata01:elk/elastic/data
    # The ports to expose from this service
    ports:
      - ${ES_PORT}:9200
    # The environment variables to set inside the container
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    # The memory limit for this service
    mem_limit: ${MEM_LIMIT}
    # The ulimits to set for this service
    ulimits:
      memlock:
        soft: -1
        hard: -1
    # The healthcheck to determine the health of this service
    healthcheck:
      test:
       [
         "CMD-SHELL",
         "curl --cacert config/certs/ca/ca.crt -u 'elastic:xxxxxx' https://localhost:9200 -k |grep -q 01"
       ]
      interval: 10s
      timeout: 10s
      retries: 120

  # The kibana service represents a Kibana instance
  kibana:
    # This service depends on the es01 service
    depends_on:
      es01:
        condition: service_healthy
    # The Docker image to use for this service
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    # The volumes to mount into this service
    volumes:
      - ./config/certs:/usr/share/kibana/config/certs
      - kibanadata: elk/kibana/data
    # The ports to expose from this service
    ports:
      - ${KIBANA_PORT}:5601
    # The environment variables to set inside the container
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - elasticsearch.ssl.certificateAuthorities=["/usr/share/kibana/config/certs/ca/ca.crt"]
      - xpack.reporting.encryptionKey=${KIBANA_PASSWORD}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=yyyyyy
    # The memory limit for this service
    mem_limit: ${MEM_LIMIT}
    # The healthcheck to determine the health of this service
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

# The volumes to create for the application
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

Removed elastic-stack-security

Added elastic-stack-security

Hi @Mubolio

Your Kibana section is incorrect see here

Wrong syntax

Hello @stephenb,

thanks for your reply, I changed the syntax of that line since I was at some point trying out different things.

When I use the correct syntax

  - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt

Still I get the same error. Is there a way to get more log information? At least of what request is doing Kibana to communicate to Elastic when fails?

As I mention before the curl command using the ca worked out for the proper credentials, so not sure what could be the problem.

I generated manually the kibana.yml, since the automatically generated one was having an incorrect url for the elastic host(not being used the one in the docker-compose.yml):

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
server.ssl.certificateAuthorities: config/certs/ca/ca.crt
elasticsearch.hosts: "http://es01:9200"

And I plaste here the logs related to auth before the error happens:

kibana_1  | [2024-09-18T09:01:44.746+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
kibana_1  | [2024-09-18T09:01:44.780+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
kibana_1  | [2024-09-18T09:01:46.466+00:00][INFO ][plugins-system.standard] Setting up [149] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,links,lens,maps,discover,reporting,canvas,fleet,logExplorer,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,metricsDataAccess,datasetQuality,aiops,ml,observabilityAIAssistant,logsShared,observabilityLogExplorer,enterpriseSearch,observability,uptime,synthetics,observabilityOnboarding,infra,upgradeAssistant,monitoring,logstash,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,assetManager,apm,ux]
kibana_1  | [2024-09-18T09:01:46.602+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: bae71735-c5de-45b4-935a-55920b5bdb2e
kibana_1  | [2024-09-18T09:01:46.857+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
kibana_1  | [2024-09-18T09:01:47.234+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
kibana_1  | [2024-09-18T09:01:47.456+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:47.456+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
kibana_1  | [2024-09-18T09:01:47.473+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:47.473+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
kibana_1  | [2024-09-18T09:01:47.596+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:47.688+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:47.695+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
kibana_1  | [2024-09-18T09:01:47.854+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:47.854+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
kibana_1  | [2024-09-18T09:01:48.548+00:00][INFO ][plugins.alerting] Registering resources for context "stack".
kibana_1  | [2024-09-18T09:01:48.677+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1  | [2024-09-18T09:01:48.678+00:00][INFO ][plugins.reporting.config] Overriding server host address "0.0.0.0" in Reporting runtime config, using "xpack.reporting.kibanaServer.hostname: localhost".
kibana_1  | [2024-09-18T09:01:49.041+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
kibana_1  | [2024-09-18T09:01:49.489+00:00][INFO ][plugins.alerting] Registering resources for context "ml.anomaly-detection".
kibana_1  | [2024-09-18T09:01:49.831+00:00][INFO ][plugins.alerting] Registering resources for context "observability.slo".
kibana_1  | [2024-09-18T09:01:49.833+00:00][INFO ][plugins.alerting] Registering resources for context "observability.threshold".
kibana_1  | [2024-09-18T09:01:49.903+00:00][INFO ][plugins.alerting] Registering resources for context "observability.uptime".
kibana_1  | [2024-09-18T09:01:50.066+00:00][INFO ][plugins.infra] Skipping initialization of Profiling endpoints because 'profilingDataAccess' plugin is not available
kibana_1  | [2024-09-18T09:01:50.067+00:00][INFO ][plugins.alerting] Registering resources for context "observability.logs".
kibana_1  | [2024-09-18T09:01:50.068+00:00][INFO ][plugins.alerting] Registering resources for context "observability.metrics".
kibana_1  | [2024-09-18T09:01:51.390+00:00][INFO ][plugins.alerting] Registering resources for context "security".
kibana_1  | [2024-09-18T09:01:51.437+00:00][INFO ][plugins.assetManager] Server is NOT enabled
kibana_1  | [2024-09-18T09:01:51.447+00:00][INFO ][plugins.alerting] Registering resources for context "observability.apm".
kibana_1  | [2024-09-18T09:01:51.631+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

Thanks.
Regards.