@ropc , Currently we have upgraded to Kibana and ES to 8.3.3. The login issue is getting intermittently. Please find below kibana logs for reference and please suggest on next steps.
Also, When checked in Kibana status page under stack monitoring section. some of the plugins status shows yellow with plugin degraded message. PFA screenshot for the same.
Would this degraded plugins status cause login issue?
Kibana log:
Aug 23 11:56:17 <Kibana-Host> kibana[472]: [2022-08-23T11:56:17.051+00:00][ERROR][plugins.taskManager] [WorkloadAggregator]: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:18 <Kibana-Host> kibana[472]: [2022-08-23T11:56:18.018+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:21 <Kibana-Host> kibana[472]: [2022-08-23T11:56:21.017+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:24 <Kibana-Host> kibana[472]: [2022-08-23T11:56:24.017+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:27 <Kibana-Host> kibana[472]: [2022-08-23T11:56:27.019+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:30 <Kibana-Host> kibana[472]: [2022-08-23T11:56:30.019+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:33 <Kibana-Host> kibana[472]: [2022-08-23T11:56:33.019+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:36 <Kibana-Host> kibana[472]: [2022-08-23T11:56:36.019+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:39 <Kibana-Host> kibana[472]: [2022-08-23T11:56:39.021+00:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
Aug 23 11:56:40 <Kibana-Host> systemd[1]: Stopped Kibana.
-- Subject: Unit kibana.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kibana.service has finished shutting down.
Aug 23 11:56:40 <Kibana-Host> systemd[1]: Started Kibana.
-- Subject: Unit kibana.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kibana.service has finished starting up.
--
-- The start-up result is done.
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:53.982+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.106+00:00][INFO ][http.server.Preboot] http server running at http://10.120.115.34:5601
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.174+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.232+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.525+00:00][INFO ][plugins-system.standard] Setting up [118] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,kibanaUsageCollection,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.546+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: a13bf5a9-8186-4da5-a7a8-3bd329a934bf
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.630+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.665+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
Aug 23 11:56:54 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:54.840+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.607+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.9.2009 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.784+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.785+00:00][INFO ][savedobjects-service] Starting saved objects migrations
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.835+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 17ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.909+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 74ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.917+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 8ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.925+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 125ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.928+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 11ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.931+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 6ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.948+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 20ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.951+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 20ms.
Aug 23 11:56:55 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:55.956+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 5ms.
Aug 23 11:56:56 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:56.058+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 102ms.
Aug 23 11:56:56 <Kibana-Host> kibana[6372]: [2022-08-23T11:56:56.499+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Aug 23 11:57:15 <Kibana-Host> kibana[6372]: [2022-08-23T11:57:15.015+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 18957ms.
Aug 23 11:57:15 <Kibana-Host> kibana[6372]: [2022-08-23T11:57:15.015+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 19215ms
Aug 23 11:57:56 <Kibana-Host> kibana[6372]: [2022-08-23T11:57:56.010+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with '[timeout_exception] Timed out waiting for completion of [Task{id=13293779, type='transport', action='indices:data/write/update/byquery', description='update-by-query [.kibana_task_manager_8.3.3_001]', parentTask=unset, startTime=1661255815945, startTimeNanos=8838582368699888}]'. Retrying attempt 1 in 2 seconds.
Aug 23 11:57:56 <Kibana-Host> kibana[6372]: [2022-08-23T11:57:56.011+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 60062ms.
Aug 23 11:58:58 <Kibana-Host> kibana[6372]: [2022-08-23T11:58:58.109+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with '[timeout_exception] Timed out waiting for completion of [Task{id=13293779, type='transport', action='indices:data/write/update/byquery', description='update-by-query [.kibana_task_manager_8.3.3_001]', parentTask=unset, startTime=1661255815945, startTimeNanos=8838582368699888}]'. Retrying attempt 2 in 4 seconds.
Aug 23 11:58:58 <Kibana-Host> kibana[6372]: [2022-08-23T11:58:58.109+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 62099ms.
Aug 23 11:59:02 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:02.117+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 4008ms.
Aug 23 11:59:02 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:02.117+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 126299ms
Aug 23 11:59:02 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:02.128+00:00][INFO ][status] Kibana is now unavailable
Aug 23 11:59:02 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:02.129+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
Aug 23 11:59:02 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:02.130+00:00][INFO ][plugins-system.standard] Starting [118] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,kibanaUsageCollection,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
Aug 23 11:59:03 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:03.100+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
Aug 23 11:59:04 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:04.924+00:00][INFO ][http.server.Kibana] http server running at http://10.120.115.34:5601
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.118+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.119+00:00][INFO ][plugins.fleet] Beginning fleet setup
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.167+00:00][INFO ][status] Kibana is now degraded (was unavailable)
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.473+00:00][INFO ][plugins.fleet] Fleet setup completed
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.482+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.862+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.862+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.863+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.863+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.863+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.863+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.864+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.873+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.875+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.877+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.878+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
Aug 23 11:59:05 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:05.880+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
Aug 23 11:59:06 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:06.133+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
Aug 23 11:59:30 <Kibana-Host> kibana[6372]: [2022-08-23T11:59:30.369+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
Aug 23 12:00:17 <Kibana-Host> kibana[6372]: [2022-08-23T12:00:17.187+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
Aug 23 12:00:30 <Kibana-Host> kibana[6372]: [2022-08-23T12:00:30.379+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
Aug 23 12:01:30 <Kibana-Host> kibana[6372]: [2022-08-23T12:01:30.384+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
Aug 23 12:01:33 <Kibana-Host> kibana[6372]: [2022-08-23T12:01:33.751+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization
Aug 23 12:01:54 <Kibana-Host> kibana[6372]: [2022-08-23T12:01:54.237+00:00][INFO ][status] Kibana is now available (was degraded)
Regards,
Avinash