Kibana container is not ready yet

Elastic Stack Kibana
1

Hi all,

After configuring a cluster using docker compose, on multiple hosts, and that it worked correctly, allowing us to create indexes correctly, we have stopped the containers and restarted them to check that everything is up correctly.

In the case of the cluster, if we point to https://XXXXX/_cluster/health?pretty the information we get back:

{
  "cluster_name" : "clp-docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Shows that the cluster runs correctly, but in the case of kibana if we point to http://YYYY:5601/api/status this is not the case:

> GET /api/status HTTP/1.1
> Host: YYYY:5601
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< x-content-type-options: nosniff
< referrer-policy: strict-origin-when-cross-origin
< permissions-policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=()
< cross-origin-opener-policy: same-origin
< content-security-policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'
< kbn-name: i-ast-clp-kib01
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 46
< Date: Wed, 05 Jun 2024 13:11:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{
  "status": {
    "overall": {
      "level":"unavailable"
     }
   }
}

Also, when trying to access the UI the same thing happens.

image
image877Ă—415 5.14 KB

We have tried to remove all containers and volumes to perform a clean boot of all services and in the case of the cluster everything seems OK except when we make a request to https://XXXX:9200/_cat/indices?pretty that returns nothing.

And in the case of kibana the same error still occurs.

On the other hand, none of the docker-compose files have been modified between the tests performed.

what could be happening?

thx!

2

Could you check the Kibana logs?

3

Sorry, I thought I had attached the container log as well, here it is:

kibana-1  | Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.13/production.html#openssl-legacy-provider
kibana-1  | {"log.level":"info","@timestamp":"2024-06-05T13:46:06.930Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.4.0","env":{"pid":7,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 5.15.0-102-generic","arch":"x64","host":"i-ast-clp-kib01","timezone":"UTC+00","runtime":"Node.js v20.11.1"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","2e3a5cd43e835baa1d596b1aa54735992259ecb9"]],"sourceValue":{"git_rev":"2e3a5cd43e835baa1d596b1aa54735992259ecb9"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.13.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.4.0"}
kibana-1  | Native global console methods have been overridden in production environment.
kibana-1  | [2024-06-05T13:46:08.563+00:00][INFO ][root] Kibana is starting
kibana-1  | [2024-06-05T13:46:08.636+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
kibana-1  | [2024-06-05T13:46:18.736+00:00][INFO ][plugins-service] The following plugins are disabled: "cloudChat,cloudExperiments,cloudFullStory,profilingDataAccess,profiling,securitySolutionServerless,serverless,serverlessObservability,serverlessSearch".
kibana-1  | [2024-06-05T13:46:18.827+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
kibana-1  | [2024-06-05T13:46:18.999+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
kibana-1  | [2024-06-05T13:46:19.072+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
kibana-1  | [2024-06-05T13:46:24.273+00:00][INFO ][plugins-system.standard] Setting up [149] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,lens,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,metricsDataAccess,aiops,links,discover,reporting,canvas,fleet,osquery,logsExplorer,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,datasetQuality,cloudSecurityPosture,cloudDefend,discoverEnhanced,maps,dataVisualizer,ml,observabilityAIAssistant,logsShared,observabilityLogsExplorer,enterpriseSearch,observability,uptime,synthetics,observabilityOnboarding,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,infra,upgradeAssistant,monitoring,logstash,assetManager,apm,ux]
kibana-1  | [2024-06-05T13:46:24.538+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 1804543d-dea3-47dd-8f9b-545f8116510d
kibana-1  | [2024-06-05T13:46:25.061+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
kibana-1  | [2024-06-05T13:46:25.781+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
kibana-1  | [2024-06-05T13:46:26.219+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:26.219+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
kibana-1  | [2024-06-05T13:46:26.255+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:26.255+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
kibana-1  | [2024-06-05T13:46:26.469+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:26.651+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:26.665+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
kibana-1  | [2024-06-05T13:46:26.970+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:26.971+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
kibana-1  | [2024-06-05T13:46:28.835+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana-1  | [2024-06-05T13:46:28.837+00:00][INFO ][plugins.reporting.config] Overriding server host address "0.0.0.0" in Reporting runtime config, using "xpack.reporting.kibanaServer.hostname: localhost".
kibana-1  | [2024-06-05T13:46:29.719+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
kibana-1  | [2024-06-05T13:46:33.247+00:00][INFO ][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]
kibana-1  | [2024-06-05T13:46:33.779+00:00][INFO ][plugins.assetManager] Server is NOT enabled
kibana-1  | [2024-06-05T13:46:34.317+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate
kibana-1  | [2024-06-05T13:46:35.140+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
kibana-1  | [2024-06-05T14:06:28.409+00:00][ERROR][plugins.ruleRegistry] Error: Timeout: it took more than 1200000ms
kibana-1  |     at Timeout._onTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:43:18)
kibana-1  |     at listOnTimeout (node:internal/timers:573:17)
kibana-1  |     at processTimers (node:internal/timers:514:7)
kibana-1  | [2024-06-05T14:06:28.411+00:00][ERROR][plugins.ruleRegistry] Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms
kibana-1  |     at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:59:13)
kibana-1  |     at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)

Previously the logs that we could see were the same except in the case of the line where it says:

kibana-1  | [2024-06-05T13:46:33.779+00:00][INFO ][plugins.assetManager] Server is NOT enabled

Which would have caught our attention, although we can't be sure that it didn't appear when it was working properly.

In addition, when searching on this always links us to posts in whose logs appear error messages with certificates, but in ours do not appear.

4

Not a Kibana expert here but AFAICS the error message indicates a bad certificate. Are you using self-signed certificates?

Did you run all the steps described here? Install Kibana with Docker | Kibana Guide [8.14] | Elastic

Specifically step 6?

5

Solution for the Kibana setup.
if you see 'missing authentication credentials' or 'Timeout' in your windows logs.
add below lines to your yml file
elasticsearch.username: 'your_elasticsearch_username'
elasticsearch.password: 'your_elasticsearch_password'

6

Hello again,

After performing several tests with the KIBANA configuration, we have finally found the problem, and it is that while in the environment of the compose file for the configuration for the certified authorities of the elastic nodes they are passed as follows: xpack.security.transport.ssl.certificate_authorities in the case of kibana they must be indicated by the key ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES

With this change Kibana can be initialized correctly.

Thank you all for your help :slight_smile:

1 Like