We are running a 3 Node ES Cluster with basic security (inter-node TLS) enabled. We installed Kibana on one of the Nodes initially and are now trying to install it on a second node in the cluster.
The /etc/kibana/kibana.yml file is
server.host: "192.168.1.120"
elasticsearch.hosts: ["https://192.168.1.120:9200"]
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
Using the elasticsearch-create-enrollment-token -s kibana returns the following error
bin/elasticsearch-create-enrollment-token -s kibana
ERROR: Failed to determine the health of the cluster.
The new Kibana installation does not load the login page, instead it displays a line of text
Kibana server is not ready yet.
The logs generated (/var/log/kibana/kibana.log) are
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:26.460+05:30","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.042+05:30","message":"Plugin \"cloudChat\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.043+05:30","message":"Plugin \"cloudExperiments\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.043+05:30","message":"Plugin \"cloudFullStory\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.043+05:30","message":"Plugin \"cloudGainsight\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.059+05:30","message":"Plugin \"profiling\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.114+05:30","message":"http server running at http://192.168.1.116:5601","log":{"level":"INFO","logger":"http.server.Preboot"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.185+05:30","message":"Setting up [1] plugins: [interactiveSetup]","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":45136},"trace":{"id":"194c92451a02f42d8dff7bc6faee6fe0"},"transaction":{"id":"263aacbb327d31f5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.208+05:30","message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading.","log":{"level":"WARN","logger":"config.deprecation"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.469+05:30","message":"Setting up [136] plugins: [usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,translations,share,screenshotMode,newsfeed,savedObjectsFinder,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,management,cloudDataMigration,advancedSettings,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,grokdebugger,console,contentManagement,bfetch,data,watcher,unifiedFieldList,savedSearch,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,embeddable,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,eventAnnotation,ecsDataQualityDashboard,dataViewFieldEditor,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,dashboard,lens,maps,aiops,dashboardEnhanced,dataViewManagement,alerting,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,discover,reporting,canvas,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,cloudDefend,exploratoryView,observability,observabilityOnboarding,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,assetManager]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.473+05:30","message":"TaskManager is identified by the Kibana UUID: dddc360d-4823-4b2d-b459-0ffaf981570a","log":{"level":"INFO","logger":"plugins.taskManager"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.490+05:30","message":"CustomBrandingService registering plugin: customBranding","log":{"level":"INFO","logger":"custom-branding-service"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.519+05:30","message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.519+05:30","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.531+05:30","message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.532+05:30","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.539+05:30","message":"Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: 2K5PS3lOiuNQ6/EMev4tbjRW0arIdxXSqk00Wv7p6TE=","log":{"level":"INFO","logger":"plugins.encryptedSavedObjects"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.545+05:30","message":"Email Service Error: Email connector not specified.","log":{"level":"INFO","logger":"plugins.notifications"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.703+05:30","message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.reporting.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.727+05:30","message":"Registered task successfully [Task: cloud_security_posture-stats_task]","log":{"level":"INFO","logger":"plugins.cloudSecurityPosture"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.730+05:30","message":"Registering resources for context \"observability.slo\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.743+05:30","message":"Registering resources for context \"observability.uptime\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.754+05:30","message":"Registering resources for context \"security\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.770+05:30","message":"Registering resources for context \"observability.logs\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.771+05:30","message":"Registering resources for context \"observability.metrics\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.811+05:30","message":"Registering resources for context \"observability.apm\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.812+05:30","message":"Asset manager plugin [tech preview] is NOT enabled","log":{"level":"INFO","logger":"plugins.assetManager"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.872+05:30","message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Debian 12 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.","log":{"level":"WARN","logger":"plugins.screenshotting.config"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:31.889+05:30",**"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception\n\tRoot causes:\n\t\tsecurity_exception: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]","log":**{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-06-22T18:24:32.208+05:30","message":"Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":45136},"trace":{"id":"046415f9540db3eda00c439e7abd9bc1"},"transaction":{"id":"7ab9e9f2498a0d99"}}
It seems Kibana is unable to retrieve information from ES even though it is on the same Node.
The strange thing about these logs is that the Version is mentioned as 8.6.1 where as the version of Kibana installed on the server is
#dpkg -l|grep kibana
ii kibana 8.8.1 amd64 Explore and visualize your Elasticsearch data
We have tried configuring Kibana to connect to the ES instance running on the same node and also on other nodes, still getting the same message on the browser.
What could be wrong here?