Kibana 8.7.0 rebooting

Elastic Stack Kibana
1

Hello guys,

I'm trying to deploy kibana v8.7.0 on k8s in order to monitore third-party elastic instance v8.7

the pod keep rebooting with the same logs...

[2025-05-07T14:46:48.622+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2025-05-07T14:46:48.628+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2025-05-07T14:46:48.629+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2025-05-07T14:46:48.629+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2025-05-07T14:46:48.672+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2025-05-07T14:46:48.875+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601/kibana
[2025-05-07T14:46:48.956+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2025-05-07T14:46:49.120+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2025-05-07T14:46:49.961+00:00][INFO ][plugins-system.standard] Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2025-05-07T14:46:50.002+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2025-05-07T14:46:50.013+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 8794e496-df50-4d0e-898e-d2aac95947ce
[2025-05-07T14:46:50.440+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:50.441+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2025-05-07T14:46:50.523+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:50.523+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2025-05-07T14:46:50.549+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:50.572+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:50.589+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2025-05-07T14:46:50.886+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:51.008+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2025-05-07T14:46:51.010+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2025-05-07T14:46:51.149+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2025-05-07T14:46:51.971+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2025-05-07T14:46:53.152+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2025-05-07T14:46:53.531+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2025-05-07T14:46:53.532+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2025-05-07T14:46:53.663+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 54ms.
[2025-05-07T14:46:53.680+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 78ms.
[2025-05-07T14:46:53.694+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 14ms.
[2025-05-07T14:46:53.704+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 41ms.
[2025-05-07T14:46:53.721+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 27ms.
[2025-05-07T14:46:53.728+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 20ms.
[2025-05-07T14:46:53.741+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 20ms.
[2025-05-07T14:46:53.742+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2025-05-07T14:46:53.743+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2025-05-07T14:46:53.743+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 141ms
[2025-05-07T14:46:53.744+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 20ms.
[2025-05-07T14:46:53.745+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2025-05-07T14:46:53.745+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 0ms.
[2025-05-07T14:46:53.746+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 137ms
[2025-05-07T14:46:53.763+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2025-05-07T14:46:53.768+00:00][INFO ][plugins-system.standard] Starting [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]

Here is my env variables:
containers:

  • name: kibana
    env:
    • name: SERVER_BASEPATH
      value: /kibana
    • name: SERVER_REWRITEBASEPATH
      value: 'true'
    • name: NODE_OPTIONS
      value: '--max-old-space-size=4096'

I've also add to the kibana.yml the lines:

elasticsearch.username: "username"
elasticsearch.password: "password"

Have you an idea on what I'm missing?

Thanks

3

Is it possible to paste what you have in your kibana.yml file here to better understand what you are missing there

4

Hello @Adebowale , thanks for your reply

I found the issue from your previous post. I have upgrade the pod resource as suggested
requests:
cpu: 1
memory: 4Gi
limits:
cpu: 2
memory: 6Gi

And everything is OK :slight_smile:

5

so it means Increasing the Resource Limits fixed the issues and this was what i posted before

The NODE_OPTIONS environment variable is set to increase memory. Ensure your Kubernetes pod has sufficient resource limits:

resources:
  limits:
    memory: "6Gi"
    cpu: "2"
  requests:
    memory: "4Gi"
    cpu: "1"
6

Yeah definitely
setting limit memory to 6G fix it

1 Like