Kibana server is not ready yet

Elastic Stack Kibana
1

Hi There,

I have recently enabled SSL for my elastic instance using out company provided SSL certificate. The elasticsearch is running fine and i can query it using https url. However when I open Kibana it says "Kibana server is not ready yet". The kibana.log says "Unable to retrieve version information from Elasticsearch nodes.".

I am in RHEL 7 and have installed an latest 7.13 RPM package.

Following is my Elastic.yml file

http.host: 0.0.0.0
http.cors.enabled : true
#http.cors.allow-origin: "*"
http.cors.allow-origin: /https?://(FQDN:[0-9]+)?/
#http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
#http.cors.allow-headers: X-Requested-With,X-Auth-Token,Content-Type,Content-Length
#http.cors.allow-credentials: true

http.port: 9200
node.name: FQDN.com
network.host: FQDN.com
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: certificate.key
xpack.security.http.ssl.certificate: certificate.crt
xpack.security.http.ssl.certificate_authorities: CompanyRootCA.crt, CompanyIssuingCA.crt
xpack.security.transport.ssl.key: certificate.key
xpack.security.transport.ssl.certificate: certificate.crt
xpack.security.transport.ssl.certificate_authorities: CompanyRootCA.crt, CompanyIssuingCA.crt
discovery.seed_hosts: [ "FQDN.com" ]
cluster.initial_master_nodes: [ "FQDN.com" ]

Following is my Kibana.yml file

server.host: "0.0.0.0"
elasticsearch.username: "username"
elasticsearch.password: "password"

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/FQDN.crt
server.ssl.key: /etc/kibana/FQDN.key

If I provide elasticsearch.ssl.certificate: /etc/elasticsearch/FQDN.crt abd elasticsearch.ssl.key: /etc/elasticsearch/FQDN.key

its not even openting. Without them Kibana is opening with proper certificate but shows "Kibana server is not ready yet".

Any help/pointer is appreciated.

Kr
Sagar

2

Can you please post the entire Kibana log.

Please also format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

3

HI @warkolm ,
Thanks for your response. Below is the entire log that I have. Please note that due to size limit of 13000 character, I had to remove some logs. But those are almost same what we have here.
Also th epoint to note here is that we don't use elastic cert util to generate certificates. We have our own company provided CA where we generate certificates. And we are using the same .crt & .key files in elastic.yml & kibana.yml. Will that be a problem?

{"type":"log","@timestamp":"2021-07-06T02:21:46-04:00","tags":["info","plugins-system"],"pid":8528,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-06T02:21:46-04:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":8528,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-06T02:22:16-04:00","tags":["warning","plugins-system"],"pid":8528,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["info","plugins-service"],"pid":12693,"message":"Plugin \"timelines\" is disabled."}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["warning","config","deprecation"],"pid":12693,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["warning","config","deprecation"],"pid":12693,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["warning","config","deprecation"],"pid":12693,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["info","plugins-system"],"pid":12693,"message":"Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeMetric,visTypeTable,visTypeVega,visTypeTagcloud,visTypeVislib,visTypeTimelion,features,licenseManagement,watcher,visTypeMarkdown,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dashboardMode,dataEnhanced,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["info","plugins","taskManager"],"pid":12693,"message":"TaskManager is identified by the Kibana UUID: 2faf4bfa-bc21-4c16-8e93-48d9db0c5221"}
{"type":"log","@timestamp":"2021-07-06T02:25:22-04:00","tags":["warning","plugins","reporting","config"],"pid":12693,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-06T02:25:23-04:00","tags":["info","plugins","monitoring","monitoring"],"pid":12693,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-06T02:25:23-04:00","tags":["info","savedobjects-service"],"pid":12693,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-06T02:25:23-04:00","tags":["error","savedobjects-service"],"pid":12693,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2021-07-06T02:31:50-04:00","tags":["info","plugins-system"],"pid":12693,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-06T02:31:50-04:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":12693,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-06T02:32:20-04:00","tags":["warning","plugins-system"],"pid":12693,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["info","plugins-service"],"pid":17077,"message":"Plugin \"timelines\" is disabled."}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["warning","config","deprecation"],"pid":17077,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["warning","config","deprecation"],"pid":17077,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["warning","config","deprecation"],"pid":17077,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["info","plugins-system"],"pid":17077,"message":"Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeVega,visTypeVislib,visTypeMetric,visTypeTimelion,features,licenseManagement,watcher,visTypeTagcloud,visTypeTable,visTypeMarkdown,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,enterpriseSearch,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["info","plugins","taskManager"],"pid":17077,"message":"TaskManager is identified by the Kibana UUID: 2faf4bfa-bc21-4c16-8e93-48d9db0c5221"}
{"type":"log","@timestamp":"2021-07-06T02:46:27-04:00","tags":["warning","plugins","reporting","config"],"pid":17077,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-06T02:46:28-04:00","tags":["info","plugins","monitoring","monitoring"],"pid":17077,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-06T02:46:28-04:00","tags":["info","savedobjects-service"],"pid":17077,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-06T02:46:28-04:00","tags":["error","savedobjects-service"],"pid":17077,"message":"Unable to retrieve version information from Elasticsearch nodes."}
4

Hi @warkolm ,

Its working now. I had to uninstall and install it again, provided elasticsearch.ssl.certificateAuthorities values in .pem format and finally server.host would be my FQDN in stead of 0.0.0.0. It started working after making above changes and thanks again for your response. I will close this thread.

BR
Sagar

5

Hi! Very strange solution. In my case it isn't work. Kibana logs:

2021-07-08T08:31:15.290819033Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["info","plugins-service"],"pid":953,"message":"Plugin \"timelines\" is disabled."}
2021-07-08T08:31:15.367802770Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","config","deprecation"],"pid":953,"message":"plugins.scanDirs is deprecated and is no longer used"}
2021-07-08T08:31:15.368009006Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","config","deprecation"],"pid":953,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
2021-07-08T08:31:15.559032645Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["info","plugins-system"],"pid":953,"message":"Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeTagcloud,visTypeTimelion,features,licenseManagement,watcher,visTypeTable,visTypeVislib,visTypeVega,visTypeMetric,visTypeMarkdown,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,dataEnhanced,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,remoteClusters,crossClusterReplication,rollup,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]"}
2021-07-08T08:31:15.560443114Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["info","plugins","taskManager"],"pid":953,"message":"TaskManager is identified by the Kibana UUID: eb36ffd6-3716-4f95-91ec-cc8a4e773d6e"}
2021-07-08T08:31:15.762508795Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","security","config"],"pid":953,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
2021-07-08T08:31:15.799680495Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","reporting","config"],"pid":953,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
2021-07-08T08:31:15.805551783Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","reporting","config"],"pid":953,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.4.2105\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
2021-07-08T08:31:15.811133268Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":953,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
2021-07-08T08:31:15.894587760Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","actions","actions"],"pid":953,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
2021-07-08T08:31:15.905195328Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":953,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
2021-07-08T08:31:15.967656783Z {"type":"log","@timestamp":"2021-07-08T08:31:15+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":953,"message":"config sourced from: production cluster"}
2021-07-08T08:31:16.164880446Z {"type":"log","@timestamp":"2021-07-08T08:31:16+00:00","tags":["info","savedobjects-service"],"pid":953,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
2021-07-08T08:31:16.232552700Z {"type":"log","@timestamp":"2021-07-08T08:31:16+00:00","tags":["error","savedobjects-service"],"pid":953,"message":"Unable to retrieve version information from Elasticsearch nodes."}

Part of docker-compose.yml with kibana settings:

  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      SERVER_HOST: kib01
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: 4InJFrn0sog4oDH57n02
      SERVER_SSL_ENABLED: "true"
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/Sub2_CA_2012.cer
      SERVER_SSL_KEY: $CERTS_DIR/elk-dkb-test1_nopass.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/elk-dkb-test1.pem
    volumes:
      - /some/path/certs:$CERTS_DIR
    networks:
      - elastic
6

Hi @Trueman
I am no export here. I would leave it for the experts to comment. However I see one difference that you have from my configuration. ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES should be in pem format and SERVER_SSL_CERTIFICATE should be crt or cer.

Hope this helps.

Br
Sagar

7

I think it is does not matter. See SSL Converter - Convert SSL Certificates to different formats

8

By pem format I mean .pem file extension. Saying because even i tried .cer/.crt. Nothing worked. However .pem worked.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.