Kibana failed to log in to the es cluster deployed by ECK, and the login timed out

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Bug Report

What did you do?
I have installed ECK via CRD 2.7 in my Kubernetes cluster (v1.24.9) . After I have deployed Elasticsearch and Kibana via the manifests shown below.
After a few minutes I can access to Kibana, the status for Kibana and Elasticsearch is in green

NAME   HEALTH   NODES   VERSION   PHASE   AGE
biot   green    3       8.7.0     Ready   7m33s
NAME    HEALTH   NODES   VERSION   AGE
biotk   green    1       8.7.0     93s

But I used account: “elastic” to click on the login and kept going around, unable to log in and when I check on the logs from Kibana I see many errors that you can check below:

[2023-04-19T05:24:28.329+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
--
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.248+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.250+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.251+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.253+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.279+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.626+00:00][INFO ][http.server.Preboot] http server running at https://0.0.0.0:5601
Wed, Apr 19 2023 1:24:56 pm | [2023-04-19T05:24:56.917+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
Wed, Apr 19 2023 1:24:57 pm | [2023-04-19T05:24:57.003+00:00][WARN ][config.deprecation] 在未来版本中，报告权限的默认工作机制会有所不同，这将影响此集群的行为。在升级之前，请将“xpack.reporting.roles.enabled”设置为“false”以采用未来的行为。
Wed, Apr 19 2023 1:24:57 pm | [2023-04-19T05:24:57.926+00:00][INFO ][plugins-system.standard] Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
Wed, Apr 19 2023 1:24:57 pm | [2023-04-19T05:24:57.955+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
Wed, Apr 19 2023 1:24:57 pm | [2023-04-19T05:24:57.964+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 71bd3d95-6841-4cd9-bc1a-1fd9ef635728
Wed, Apr 19 2023 1:24:58 pm | [2023-04-19T05:24:58.205+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: MzXZMxxomAkNAzoVZyWAowzuHRQ6APT5AUjpVVpn7EY=
Wed, Apr 19 2023 1:24:58 pm | [2023-04-19T05:24:58.238+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
Wed, Apr 19 2023 1:24:58 pm | [2023-04-19T05:24:58.474+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
Wed, Apr 19 2023 1:24:58 pm | [2023-04-19T05:24:58.535+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
Wed, Apr 19 2023 1:24:59 pm | [2023-04-19T05:24:59.072+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
Wed, Apr 19 2023 1:24:59 pm | [2023-04-19T05:24:59.853+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.368+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.370+00:00][INFO ][savedobjects-service] Starting saved objects migrations
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.564+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception
Wed, Apr 19 2023 1:25:01 pm | Caused by:
Wed, Apr 19 2023 1:25:01 pm | index_not_found_exception: no such index [.security]
Wed, Apr 19 2023 1:25:01 pm | Root causes:
Wed, Apr 19 2023 1:25:01 pm | index_not_found_exception: no such index [.security]'. Retrying attempt 1 in 2 seconds.
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.565+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> INIT. took: 71ms.
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.596+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 122ms.
Wed, Apr 19 2023 1:25:01 pm | [2023-04-19T05:25:01.733+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Wed, Apr 19 2023 1:25:02 pm | [2023-04-19T05:25:02.469+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
Wed, Apr 19 2023 1:25:02 pm | Caused by:
Wed, Apr 19 2023 1:25:02 pm | index_not_found_exception: no such index [.security]
Wed, Apr 19 2023 1:25:02 pm | Root causes:
Wed, Apr 19 2023 1:25:02 pm | index_not_found_exception: no such index [.security]
Wed, Apr 19 2023 1:25:03 pm | [2023-04-19T05:25:03.583+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception
Wed, Apr 19 2023 1:25:03 pm | Caused by:
Wed, Apr 19 2023 1:25:03 pm | index_not_found_exception: no such index [.security]
Wed, Apr 19 2023 1:25:03 pm | Root causes:
Wed, Apr 19 2023 1:25:03 pm | index_not_found_exception: no such index [.security]'. Retrying attempt 2 in 4 seconds.
Wed, Apr 19 2023 1:25:03 pm | [2023-04-19T05:25:03.584+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> INIT. took: 2019ms.
Wed, Apr 19 2023 1:25:07 pm | [2023-04-19T05:25:07.597+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception
Wed, Apr 19 2023 1:25:07 pm | Caused by:
Wed, Apr 19 2023 1:25:07 pm | index_not_found_exception: no such index [.security]
Wed, Apr 19 2023 1:25:07 pm | Root causes:
Wed, Apr 19 2023 1:25:07 pm | index_not_found_exception: no such index [.security]'. Retrying attempt 3 in 8 seconds.
Wed, Apr 19 2023 1:25:07 pm | [2023-04-19T05:25:07.599+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> INIT. took: 4014ms.
Wed, Apr 19 2023 1:25:15 pm | [2023-04-19T05:25:15.659+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 8062ms.
Wed, Apr 19 2023 1:26:18 pm | [2023-04-19T05:26:18.641+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 62982ms.
Wed, Apr 19 2023 1:26:30 pm | [2023-04-19T05:26:30.301+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 88705ms.
Wed, Apr 19 2023 1:26:40 pm | [2023-04-19T05:26:40.981+00:00][INFO ][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 22340ms.
Wed, Apr 19 2023 1:26:40 pm | [2023-04-19T05:26:40.982+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 99489ms
Wed, Apr 19 2023 1:26:55 pm | [2023-04-19T05:26:55.650+00:00][INFO ][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 25348ms.
Wed, Apr 19 2023 1:26:55 pm | [2023-04-19T05:26:55.651+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 114176ms
Wed, Apr 19 2023 1:26:55 pm | [2023-04-19T05:26:55.694+00:00][INFO ][status] Kibana is now unavailable
Wed, Apr 19 2023 1:26:55 pm | [2023-04-19T05:26:55.697+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
Wed, Apr 19 2023 1:26:55 pm | [2023-04-19T05:26:55.702+00:00][INFO ][plugins-system.standard] Starting [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
Wed, Apr 19 2023 1:26:57 pm | [2023-04-19T05:26:57.834+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
Wed, Apr 19 2023 1:26:57 pm | [2023-04-19T05:26:57.888+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
Wed, Apr 19 2023 1:26:57 pm | [2023-04-19T05:26:57.906+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Internal collection for Kibana monitoring is disabled per configuration.
Wed, Apr 19 2023 1:27:01 pm | [2023-04-19T05:27:01.965+00:00][INFO ][http.server.Kibana] http server running at https://0.0.0.0:5601
Wed, Apr 19 2023 1:27:02 pm | [2023-04-19T05:27:02.089+00:00][INFO ][status] Kibana is now degraded (was unavailable)
Wed, Apr 19 2023 1:27:02 pm | [2023-04-19T05:27:02.496+00:00][INFO ][plugins.fleet] Beginning fleet setup
Wed, Apr 19 2023 1:27:18 pm | [2023-04-19T05:27:18.791+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.195+00:00][ERROR][plugins.fleet] **Error scheduling task, received error: TimeoutError: Request timed out**
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.199+00:00][ERROR][plugins.fleet.fleet:check-deleted-files-task:1.0.1] Error scheduling task, received error: TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.204+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-saved-queries:1.1.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.207+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-packs:1.1.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.212+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-configs:1.1.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.215+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:endpoint-meta-telemetry:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.218+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-prebuilt-rule-alerts:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.221+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-filterlist-artifact:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.244+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-timelines:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.257+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:endpoint-diagnostics:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.261+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-detection-rules:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.263+00:00][ERROR][plugins.actions] Error scheduling Actions-cleanup_failed_action_executions, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.265+00:00][ERROR][plugins.apm] Failed to schedule APM source map migration
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.266+00:00][ERROR][plugins.apm] TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:27:32 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:27:32 pm | at runNextTicks (node:internal/process/task_queues:65:3)
Wed, Apr 19 2023 1:27:32 pm | at listOnTimeout (node:internal/timers:528:9)
Wed, Apr 19 2023 1:27:32 pm | at processTimers (node:internal/timers:502:7)
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:27:32 pm | at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.283+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-lists:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.292+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-configuration:1.0.0]: error scheduling task, received Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.519+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:27:32 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:27:32 pm | at ClientTraced.DeleteApi [as delete] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/delete.js:36:12)
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.549+00:00][ERROR][plugins.security.session] Failed to schedule session index cleanup task: Request timed out
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.617+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:27:32 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:27:32 pm | at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.936+00:00][WARN ][plugins.fleet] Fleet setup failed
Wed, Apr 19 2023 1:27:32 pm | [2023-04-19T05:27:32.937+00:00][WARN ][plugins.fleet] TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:27:32 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:27:32 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:27:32 pm | at ClientTraced.IndexApi [as index] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/index.js:51:12)
Wed, Apr 19 2023 1:27:33 pm | [2023-04-19T05:27:33.006+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
Wed, Apr 19 2023 1:27:48 pm | [2023-04-19T05:27:48.800+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
Wed, Apr 19 2023 1:27:48 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:27:48 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:27:48 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:27:48 pm | at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
Wed, Apr 19 2023 1:28:02 pm | [2023-04-19T05:28:02.301+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] TimeoutError: Request timed out
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:28:02 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:02 pm | at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
Wed, Apr 19 2023 1:28:02 pm | [2023-04-19T05:28:02.311+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] TimeoutError: Request timed out
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:28:02 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:02 pm | at runNextTicks (node:internal/process/task_queues:65:3)
Wed, Apr 19 2023 1:28:02 pm | at listOnTimeout (node:internal/timers:528:9)
Wed, Apr 19 2023 1:28:02 pm | at processTimers (node:internal/timers:502:7)
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:02 pm | at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
Wed, Apr 19 2023 1:28:02 pm | [2023-04-19T05:28:02.319+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] TimeoutError: Request timed out
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:28:02 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:02 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:02 pm | at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
Wed, Apr 19 2023 1:28:02 pm | [2023-04-19T05:28:02.716+00:00][ERROR][plugins.security.session] Failed to schedule session index cleanup task: Request timed out
Wed, Apr 19 2023 1:28:03 pm | [2023-04-19T05:28:03.013+00:00][ERROR][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] EndpointError: Error scheduling task, received Request timed out
Wed, Apr 19 2023 1:28:03 pm | at ManifestTask.start (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/task.js:49:27)
Wed, Apr 19 2023 1:28:03 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.567+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
Wed, Apr 19 2023 1:28:24 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:28:24 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:24 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:24 pm | at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.697+00:00][WARN ][plugins.fleet] Failure to install package [synthetics]: [TimeoutError: Request timed out]
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.699+00:00][ERROR][plugins.fleet] uninstalling synthetics-0.11.8 after error installing: [TimeoutError: Request timed out]
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.719+00:00][ERROR][plugins.fleet] failed to uninstall or rollback package after installation error Error: synthetics is not installed
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.724+00:00][ERROR][plugins.synthetics] TimeoutError: 安装 synthetics 0.11.8 时出错: Request timed out
Wed, Apr 19 2023 1:28:24 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:28:24 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:24 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:24 pm | at ClientTraced.IndexApi [as index] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/index.js:51:12)
Wed, Apr 19 2023 1:28:24 pm | [2023-04-19T05:28:24.726+00:00][WARN ][plugins.synthetics] IndexTemplateInstallationError: Failed to install synthetics index templates.
Wed, Apr 19 2023 1:28:24 pm | at SyntheticsService.setupIndexTemplates (/usr/share/kibana/x-pack/plugins/synthetics/server/synthetics_service/synthetics_service.js:99:24)
Wed, Apr 19 2023 1:28:24 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:26 pm | [2023-04-19T05:28:26.755+00:00][WARN ][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying...
Wed, Apr 19 2023 1:28:26 pm | [2023-04-19T05:28:26.757+00:00][WARN ][plugins.apm] ResponseError: resource_already_exists_exception
Wed, Apr 19 2023 1:28:26 pm | Root causes:
Wed, Apr 19 2023 1:28:26 pm | resource_already_exists_exception: index [.apm-agent-configuration/hkvdTjRSSf-sottzuPXsKQ] already exists
Wed, Apr 19 2023 1:28:26 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
Wed, Apr 19 2023 1:28:26 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:26 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:26 pm | at Indices.create (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:191:16)
Wed, Apr 19 2023 1:28:26 pm | at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:39:12
Wed, Apr 19 2023 1:28:26 pm | at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
Wed, Apr 19 2023 1:28:26 pm | [2023-04-19T05:28:26.768+00:00][WARN ][plugins.apm] Could not create index: '.apm-custom-link'. Retrying...
Wed, Apr 19 2023 1:28:26 pm | [2023-04-19T05:28:26.769+00:00][WARN ][plugins.apm] ResponseError: resource_already_exists_exception
Wed, Apr 19 2023 1:28:26 pm | Root causes:
Wed, Apr 19 2023 1:28:26 pm | resource_already_exists_exception: index [.apm-custom-link/DS1Kj1SQR225uELRR2_ymg] already exists
Wed, Apr 19 2023 1:28:26 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
Wed, Apr 19 2023 1:28:26 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:28:26 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:28:26 pm | at Indices.create (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:191:16)
Wed, Apr 19 2023 1:28:26 pm | at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:39:12
Wed, Apr 19 2023 1:28:26 pm | at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
Wed, Apr 19 2023 1:28:32 pm | [2023-04-19T05:28:32.960+00:00][ERROR][plugins.security.session] Failed to schedule session index cleanup task: Request timed out

about login

[2023-04-19T05:29:40.414+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
--
Wed, Apr 19 2023 1:30:03 pm | [2023-04-19T05:30:03.653+00:00][ERROR][plugins.security.session] Failed to schedule session index cleanup task: Request timed out
Wed, Apr 19 2023 1:30:28 pm | [2023-04-19T05:30:28.025+00:00][WARN ][plugins.apm] Could not create index: '.apm-custom-link'. Retrying...
Wed, Apr 19 2023 1:30:28 pm | [2023-04-19T05:30:28.026+00:00][WARN ][plugins.apm] TimeoutError: Request timed out
Wed, Apr 19 2023 1:30:28 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:30:28 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:30:28 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:30:28 pm | at Indices.putMapping (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:910:16)
Wed, Apr 19 2023 1:30:28 pm | at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:35:36
Wed, Apr 19 2023 1:30:28 pm | at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
Wed, Apr 19 2023 1:30:28 pm | [2023-04-19T05:30:28.034+00:00][WARN ][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying...
Wed, Apr 19 2023 1:30:28 pm | [2023-04-19T05:30:28.035+00:00][WARN ][plugins.apm] TimeoutError: Request timed out
Wed, Apr 19 2023 1:30:28 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
Wed, Apr 19 2023 1:30:28 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:30:28 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:30:28 pm | at Indices.putMapping (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:910:16)
Wed, Apr 19 2023 1:30:28 pm | at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:35:36
Wed, Apr 19 2023 1:30:28 pm | at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
Wed, Apr 19 2023 1:30:34 pm | [2023-04-19T05:30:34.273+00:00][ERROR][plugins.security.session] Failed to schedule session index cleanup task: Request timed out
Wed, Apr 19 2023 1:30:40 pm | [2023-04-19T05:30:40.632+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"at least one primary shard for the index [.security-profile-8] is unavailable"}],"type":"unavailable_shards_exception","reason":"at least one primary shard for the index [.security-profile-8] is unavailable"},"status":503}.
Wed, Apr 19 2023 1:30:40 pm | [2023-04-19T05:30:40.634+00:00][ERROR][http] ResponseError: unavailable_shards_exception
Wed, Apr 19 2023 1:30:40 pm | Root causes:
Wed, Apr 19 2023 1:30:40 pm | unavailable_shards_exception: at least one primary shard for the index [.security-profile-8] is unavailable
Wed, Apr 19 2023 1:30:40 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
Wed, Apr 19 2023 1:30:40 pm | at processTicksAndRejections (node:internal/process/task_queues:96:5)
Wed, Apr 19 2023 1:30:40 pm | at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
Wed, Apr 19 2023 1:30:40 pm | at Security.activateUserProfile

What did you expect to see?
No errors in Kibana and the platform working correctly and i can login success!!
What did you see instead? Under which circumstances?
A lot of time-out errors
Environment

$ kubectl version
[root@k8s-master-01 elastic]# kubectl version -o=yaml
clientVersion:
  buildDate: "2022-12-08T10:15:09Z"
  compiler: gc
  gitCommit: 9710807c82740b9799453677c977758becf0acbb
  gitTreeState: clean
  gitVersion: v1.24.9
  goVersion: go1.18.9
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-12-08T10:08:06Z"
  compiler: gc
  gitCommit: 9710807c82740b9799453677c977758becf0acbb
  gitTreeState: clean
  gitVersion: v1.24.9
  goVersion: go1.18.9
  major: "1"
  minor: "24"
  platform: linux/amd64
  • Resource definition:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: biot
  namespace: elk
spec:
  version: 8.7.0
  nodeSets:
  - name: node
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            memory: 2Gi
            cpu: 2
            storage: 6Gi
          limits:
            memory: 4Gi
            cpu: 4
        storageClassName: nfs-client
    podTemplate:
     spec:
       initContainers:
       - name: sysctl
         securityContext:
           privileged: true
           runAsUser: 0
         command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
  http:
    service:
      spec:
        type: NodePort
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: biotk
  namespace: elk
spec:
  version: 8.7.0
  count: 1
  elasticsearchRef:
    name: biot
    namespace: elk
  config:
    monitoring.kibana.collection.enabled: false
    monitoring.ui.ccs.enabled: false
    i18n.locale: "zh-CN"
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 2Gi
            cpu: 2
          limits:
            memory: 2.5Gi
            cpu: 4
  http:
    service:
      spec:
        type: NodePort
2

What is in the Elasticsearch logs?

3

kibana_logs

[2023-04-25T03:27:27.162+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[2023-04-25T03:27:38.256+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: Request timed out
[2023-04-25T03:27:53.219+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2023-04-25T03:27:53.361+00:00][INFO ][status] Kibana is now degraded (was available)
[2023-04-25T03:28:19.431+00:00][ERROR][plugins.reporting.store] Error in start phase
[2023-04-25T03:28:19.432+00:00][ERROR][plugins.reporting.store] undefined
[2023-04-25T03:28:19.433+00:00][ERROR][plugins.reporting] Error in Reporting start, reporting may not function properly
[2023-04-25T03:28:19.434+00:00][ERROR][plugins.reporting] TimeoutError: Request timed out
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
at Ilm.putLifecycle (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ilm.js:201:16)
at IlmPolicyManager.createIlmPolicy (/usr/share/kibana/x-pack/plugins/reporting/server/lib/store/ilm_policy_manager/ilm_policy_manager.js:46:5)
at ReportingStore.start (/usr/share/kibana/x-pack/plugins/reporting/server/lib/store/store.js:154:7)
at /usr/share/kibana/x-pack/plugins/reporting/server/plugin.js:114:7
[2023-04-25T03:28:23.225+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out

es-logs

{"@timestamp":"2023-04-25T03:23:33.763Z", "log.level": "WARN", "message":"path: /_security/privilege/kibana-.kibana, params: {application=kibana-.kibana}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[biot-es-node-0][transport_worker][T#1]","log.logger":"rest.suppressed","trace.id":"51ef19310d1ef7c40345b907feaf0e49","elasticsearch.cluster.uuid":"feFKXswsS4W2im--R-34_Q","elasticsearch.node.id":"opw8cRaUQbOKHfKAq1jbbg","elasticsearch.node.name":"biot-es-node-0","elasticsearch.cluster.name":"biot","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"at least one primary shard for the index [.security-7] is unavailable","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:138)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStore.innerGetPrivileges(NativePrivilegeStore.java:188)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStore.getPrivileges(NativePrivilegeStore.java:169)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.privilege.TransportGetPrivilegesAction.doExecute(TransportGetPrivilegesAction.java:55)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.privilege.TransportGetPrivilegesAction.doExecute(TransportGetPrivilegesAction.java:31)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.TransportAction$RequestF
ilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:433)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:986)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:952)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$7(AuthorizationService.java:447)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:183)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:437)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:413)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:314)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.ContextPreservingA
ctionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:149)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:299)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:216)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:191)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.auth
z.AuthorizationService.authorize(AuthorizationService.java:316)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:132)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:203)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.rest.action.privilege.RestGetPrivilegesAction.lambda$innerP
repareRequest$0(RestGetPrivilegesAction.java:72)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:123)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:113)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:107)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:132)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:440)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:346)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325)\n\tat org.elasticsearch.securit
y@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:202)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:121)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:188)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.ServiceAccountAuthenticator.lambda$authenticate$0(ServiceAccountAuthenticator.java:55)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.ServiceAccountService.lambda$authenticateToken$0(ServiceAccountService.java:136)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.lambda$authenticateWithCache$1(CachingServiceAccountTokenStore.java:100)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:510)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(Listenabl
eFuture.java:111)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticateWithCache(CachingServiceAccountTokenStore.java:98)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticate(CachingServiceAccountTokenStore.java:82)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.CompositeServiceAccountTokenStore.lambda$authenticate$0(CompositeServiceAccountTokenStore.java:37)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.CompositeServiceAccountTokenStore.authenticate(CompositeServiceAccountTokenStore.java:44)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.service.ServiceAccountService.authenticateToken(ServiceAccountService.java:134)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.ServiceAccountAuthenticator.authenticate(ServiceAccountAuthenticator.java:53)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180)\n\tat org.elasticsearch.xcore@8.7.0/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)\n\tat org.el
asticsearch.security@8.7.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)\n\tat org.elasticsearch.security@8.7.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:101)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:402)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:532)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353)\n\tat org.elasticsearch.transport.netty4@8.7.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)\n\tat org.elasticsearch.transport.netty4@8.7.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHand
lerContext.java:444)\n\tat io.n
etty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.jav
a:412)\n\tat io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)\n\tat io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)\n\tat io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\tat io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.net
ty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.86.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.86.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n"}
4

Unrelated to your issue but that resource request statement is not correct I think. These are the resource requests for your volumeClaimTemplate and should only contain a storage request.

The main issue seems to be that a primary shard for the .security index is unavailable:

at least one primary shard for the index [.security-7] is unavailable

So the question you have to answer is why? I wonder if it is to do with the kind of storage you have chosen. Can you try with a fresh cluster using another form of storage just to rule out storage issues? Are other indices also unavailable?

5

My problem is that kibana cannot log in and keeps going round. The status of elasicsearch cluster is normal and I can also read and write data through api,I'll try another storage method

20230418100531
20230418100531531×567 17.5 KB

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.