Kibana degraded, available on Kubernetes

PustyB · March 6, 2023, 9:37am

Hey
I've had a problem with Kibana for a while now. I installed Elastic Stack on kubernetes using the operator. At first everything was fine, however, since some time in the kibana logs it started throwing such errors:

[2023-03-02T13:16:40.675+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:16:44.881+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:17:20.634+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:17:25.298+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:17:35.542+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:17:55.151+00:00][ERROR][plugins.taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity
[2023-03-02T13:18:27.544+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:18:55.305+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:19:31.595+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:19:55.261+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[2023-03-02T13:19:57.839+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:20:02.785+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:20:49.150+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:21:07.168+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:21:21.326+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:21:25.409+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:21:46.257+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:22:27.544+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:22:59.962+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:23:27.224+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:23:37.087+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:23:55.817+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:24:18.779+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2023-03-02T13:24:19.179+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:24:43.345+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:25:05.725+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2023-03-02T13:25:07.881+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:26:37.459+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:26:52.120+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-02T13:27:27.548+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-02T13:27:40.118+00:00][INFO ][status] Kibana is now degraded (was available)

Unfortunately I can't trace the cause of these errors, has anyone perhaps had a similar problem?

warkolm · March 7, 2023, 10:44pm

Welcome to our community!

Is there more to the log before this? Is Elasticsearch responding?

PustyB · March 12, 2023, 9:07am

[2023-03-12T09:00:03.048+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-03-12T09:00:38.996+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-03-12T09:00:38.998+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-03-12T09:00:38.999+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-03-12T09:00:38.999+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-03-12T09:00:39.019+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-03-12T09:00:39.231+00:00][INFO ][http.server.Preboot] http server running at https://0.0.0.0:5601
[2023-03-12T09:00:39.342+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-03-12T09:00:39.439+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-03-12T09:00:40.050+00:00][INFO ][plugins-system.standard] Setting up [127] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,fieldFormats,expressions,screenshotting,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,osquery,observability,aiops,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-03-12T09:00:40.089+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: xxx
[2023-03-12T09:00:40.366+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: xxx
[2023-03-12T09:00:40.405+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-03-12T09:00:40.612+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-03-12T09:00:40.749+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-03-12T09:00:40.847+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-03-12T09:00:42.428+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-03-12T09:00:42.745+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-03-12T09:00:42.747+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-03-12T09:00:42.842+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 51ms.
[2023-03-12T09:00:42.848+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 49ms.
[2023-03-12T09:00:42.853+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 12ms.
[2023-03-12T09:00:42.859+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 11ms.
[2023-03-12T09:00:42.884+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 25ms.
[2023-03-12T09:00:42.889+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 36ms.
[2023-03-12T09:00:42.894+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 10ms.
[2023-03-12T09:00:42.896+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 2ms.
[2023-03-12T09:00:42.897+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2023-03-12T09:00:42.900+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 101ms
[2023-03-12T09:00:42.904+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 14ms.
[2023-03-12T09:00:42.904+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2023-03-12T09:00:42.905+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2023-03-12T09:00:42.906+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 116ms
[2023-03-12T09:00:42.918+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-03-12T09:00:42.920+00:00][INFO ][plugins-system.standard] Starting [127] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,fieldFormats,expressions,screenshotting,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,osquery,observability,aiops,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-03-12T09:00:46.667+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
[2023-03-12T09:00:46.669+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 5m
[2023-03-12T09:00:46.817+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2023-03-12T09:00:50.940+00:00][INFO ][http.server.Kibana] http server running at https://0.0.0.0:5601
[2023-03-12T09:00:51.068+00:00][INFO ][status] Kibana is now degraded
[2023-03-12T09:00:51.247+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2023-03-12T09:00:51.249+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2023-03-12T09:00:51.337+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2023-03-12T09:00:51.338+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2023-03-12T09:00:51.338+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2023-03-12T09:00:51.339+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2023-03-12T09:00:51.339+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2023-03-12T09:00:51.340+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2023-03-12T09:00:51.340+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2023-03-12T09:00:51.359+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2023-03-12T09:00:51.371+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2023-03-12T09:00:51.372+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2023-03-12T09:00:51.506+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2023-03-12T09:00:51.509+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2023-03-12T09:00:52.796+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-03-12T09:00:54.266+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2023-03-12T09:00:55.248+00:00][INFO ][plugins.fleet] Fleet setup completed
[2023-03-12T09:00:55.268+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2023-03-12T09:00:55.768+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2023-03-12T09:01:03.731+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-12T09:01:36.300+00:00][INFO ][plugins.synthetics] Installed synthetics index templates
[2023-03-12T09:01:38.564+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-12T09:01:45.483+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-03-12T09:02:36.227+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-03-12T09:02:42.463+00:00][INFO ][status] Kibana is now available (was degraded)

Yes, Elasticsearch is responding. However, as you can see in the photo below, every now and then the connection to the elastic is broken.

warkolm · March 19, 2023, 11:26pm

Thanks, there's not a lot there that's helpful unfortunately.

Is there anything in the Elasticsearch logs at around this time?

system · April 16, 2023, 11:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.