[error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. [savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK

mujtabah · July 30, 2022, 11:53am

Hi guys i have a running cluster with 5 nodes(3master and 2 data nodes). i have done rolling upgrade of elastic for all the 5 nodes from v7.10.0 to v7.16.2 and it was successfully upgraded and all the 5 nodes are in cluster and cluster health is yellow. then i tried to do standard upgrade for kibana of all 5 nodes from v7.10.0 to v7.16.0. After i done upgrade for kibana , then i run kibana batch file parallel in all 5 nodes. iam getting this error.

PS C:\kibana-7.16.2-windows-x86_64\bin> .\kibana.bat
log [15:35:04.002] [info][plugins-service] Plugin "metricsEntities" is disabled.
log [15:35:04.185] [info][server][Preboot][http] http server running at https://0.0.0.0:5601/report
log [15:35:04.307] [warning][config][deprecation] "server.cors" is deprecated and has been replaced by "server.cors.enabled"
log [15:35:04.309] [warning][config][deprecation] Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format.
log [15:35:04.311] [warning][config][deprecation] Use Kibana application privileges to grant reporting privileges. Using "xpack.reporting.roles.allow" to grant reporting privileges is deprecated. The "xpack.reporting.roles.enabled" setting will default to false in a future release.
log [15:35:04.312] [warning][config][deprecation] User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout.
log [15:35:04.313] [warning][config][deprecation] Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout.
log [15:35:04.569] [info][plugins-system][standard] Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]
log [15:35:04.611] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 7d35f6fb-daed-4ee1-adb2-fcff2bde54d5
log [15:35:04.871] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:04.921] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:04.999] [warning][encryptedSavedObjects][plugins] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.038] [warning][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.069] [warning][alerting][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.112] [info][plugins][ruleRegistry] Installing common resources shared between all indices
log [15:35:06.268] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox.
log [15:35:10.111] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [15:35:10.113] [info][savedobjects-service] Starting saved objects migrations
log [15:35:11.219] [info][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 26ms.
log [15:35:11.225] [info][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 6ms.
log [15:35:11.245] [info][savedobjects-service] [.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 20ms.
log [15:35:11.251] [info][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 6ms.
log [15:35:11.267] [info][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 77ms.
log [15:35:11.270] [info][savedobjects-service] [.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 19ms.
log [15:35:11.274] [info][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 7ms.
log [15:35:11.284] [info][savedobjects-service] [.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 14ms.
log [15:35:11.288] [info][savedobjects-service] [.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 14ms.
log [15:35:11.291] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 7ms.
log [15:35:11.293] [info][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 5ms.
log [15:35:11.303] [info][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 10ms.
log [15:35:11.313] [info][savedobjects-service] [.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 10ms.
log [15:35:11.320] [info][savedobjects-service] [.kibana_task_manager] Starting to process 6 documents.
log [15:35:11.321] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 29ms.
log [15:35:11.328] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 8ms.
log [15:35:11.332] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 19ms.
log [15:35:11.665] [info][savedobjects-service] [.kibana] Starting to process 48 documents.
log [15:35:11.666] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 333ms.
log [15:35:11.737] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 72ms.
log [15:37:11.403] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
log [15:37:11.405] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120075ms.
log [15:37:12.104] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
log [15:37:12.105] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120367ms.
log [15:39:13.490] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
log [15:39:13.491] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 122087ms.
log [15:39:14.490] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
log [15:39:14.491] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 122386ms.
log [15:41:17.544] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
log [15:41:17.546] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 124054ms.
log [15:41:18.841] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
log [15:41:18.842] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 124351ms.

mujtabah · July 30, 2022, 2:12pm

And kibana page shows : kibana server is not yet ready

ropc · July 31, 2022, 1:40pm

Hi @mujtabah

Based on the logs you shared, it seems that the bulk APIs used as part of the Kibana saved objects migration are timing-out.

A few things worth checking here:

a) Is the cluster under heavy load? If so, you may need to look into reducing the pressure so that the saved objects migration can complete. If you have monitoring enabled, you may want to look into the Stack Monitoring data and verify the CPU, heap usage as well as the requests latency. If monitoring data is not available, you could look into using GET _cat/nodes?v and GET _cat/thread_pool to check the CPU, heap metrics as well as the requests in the thread pool queues.

b) You mentioned that the cluster is yellow - which means that replica shards are not assigned. Is there any allocation issue? To check this, you could run the following APIs:

GET _cat/indices?v&health=yellow
GET _cluster/allocation/explain
GET _cat/allocation?v

I hope that helps.

mujtabah · July 31, 2022, 3:17pm

C:\Users\Administrator> curl -u ":" -XGET https://192.168.100.211:9200/_cat/allocation?v --ssl-no-revoke
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
9 256.8kb 19.9gb 59.4gb 79.3gb 25 192.168.100.211 192.168.100.211 ICA-DB1
5 2.7mb 20.1gb 59.2gb 79.3gb 25 192.168.100.212 192.168.100.212 ICA-DB2
8 664.1kb 22.7gb 56.6gb 79.3gb 28 192.168.100.215 192.168.100.215 ICA-DB5
5 38mb 23.1gb 56.2gb 79.3gb 29 192.168.100.214 192.168.100.214 ICA-DB4
6 12mb 19.4gb 59.9gb 79.3gb 24 192.168.100.213 192.168.100.213 ICA-DB3
33 UNASSIGNED

mujtabah · July 31, 2022, 3:20pm

C:\Users\Administrator> health status index yellow open event_2022-07-29 yellow open event_2022-07-28 yellow open configuration_user_roles yellow open configuration_module yellow open configuration_object_classes yellow open configuration_user_accounts yellow open statistics_2022-07-29 yellow open .geoip_databases yellow open configuration_schedules yellow open statistics_2022-07-28 yellow open .apm-custom-link yellow open .kibana_task_manager_1 yellow open .kibana_task_manager_ yellow open .kibana_7.16.2_reindex_temp yellow open event_2022-07-30 yellow open event_2022-07-31 yellow open configuration_rule_assignments yellow open configuration_scene_regions yellow open configuration_settings yellow open .apm-agent-configuration yellow open configuration_site_maps yellow open .kibana_1 yellow open .tasks yellow open .security-7 yellow open configuration_cameras yellow open .kibana-event-log-7.10.0-000001 yellow open statistics_2022-07-30 yellow open .async-search yellow open statistics_2022-07-31 curl -u "xyz:xyz" -XGET https://192.168.100.211:9200/_cat/indices?v --ssl-no-revoke
uuid pri rep docs.count docs.deleted store.size pri.store.size
JqutIZl7S-26RSx8z83KCA 1 1 1677 121 892.4kb 892.4kb
PFkKredWT6O2HitNzk7q6w 1 1 996 194 492.8kb 492.8kb
S9iBwHaQQJukKTSyzOzbrg 1 1 3 0 19.3kb 19.3kb
ehk4ujOZTuWyOyzGJcKl4w 1 1 0 0 261b 261b
97HjtkloSK6IBMlXK09gtA 1 1 28 0 16.1kb 16.1kb
76NbB3EySdyhOl_9s53wRg 1 1 1 0 7.6kb 7.6kb
bxhviszSR9-scOcM1TrjBg 1 1 6038 0 928.4kb 928.4kb
YpmhMqGLQ36S4LUjTomnRg 1 1 40 0 37.6mb 37.6mb
4xxrSz54RCCYfNAEMpLTXg 1 1 5 0 19.2kb 19.2kb
epE3s_gCS5qX51UGLWBG5w 1 1 718 0 155kb 155kb
Tuvz2h4RTQGDaXx_-ikyWQ 1 1 0 0 261b 261b
YGsI3faBRICfAcENTDv1nQ 1 1 6 2 26.7kb 26.7kb
7.16.2_reindex_temp KjWqa-yvQXa3tndo2igJ2g 1 1 0 0 226b 226b
in9OZrUrQAONz4gnZEloVg 1 1 0 0 226b 226b
ZRb_wSj3QlCsURaUAbXyww 1 1 980 45 337.7kb 337.7kb
tktX4EW2QXaB-s9NW8f6Ig 1 1 610 64 286.8kb 286.8kb
ptm8UcdOT8iryNPxHr2eQg 1 1 1 0 8.3kb 8.3kb
wB-KYU70Tfevj_yWaQA_Mg 1 1 22 0 15.2kb 15.2kb
F2npeTg_SumJdE1xpn2uyg 1 1 3 0 100.4kb 100.4kb
C-4gPgD9SEiyzggVk5CcMA 1 1 0 0 261b 261b
8BTMqTWJQJyu1JUqd8Wesw 1 1 1 0 6.2kb 6.2kb
K9ogKSo5QrizryIMNdOutA 1 1 48 6 10.4mb 10.4mb
Z6Ls_Ph4SoS_9wuGWEzvUg 1 1 668 0 373.2kb 373.2kb
er8qFGYmQoyBqMKTqZEe-Q 1 1 46 46 283.1kb 283.1kb
RdBYvkMTQgW8hcNGL19qOw 1 1 2 0 15.3kb 15.3kb
N8pYyMgzSHaIwyCquKrJKw 1 1 49 0 19kb 19kb
YEUIFq3_S8ym4ChxKNPDYw 1 1 7756 0 1.1mb 1.1mb
ZcSDLfJERRuiKTHHOrzBTA 1 1 0 0 252b 252b
lMXJiBZIQqCpJOnzHnO3uA 1 1 5 0 358kb 358kb

mujtabah · July 31, 2022, 3:22pm

C:\Users\Administrator> curl -u "xyz:xyz" -XGET https://192.168.100.211:9200/_cluster/allocation/explain --ssl-no-revoke
{"note":"No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.","index":".apm-agent-configuration","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2022-07-30T13:31:22.109Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"2J1nm1GcRP6ofx3uYOorcA","node_name":"ICA-DB5","transport_address":"192.168.100.215:9300","node_attributes":{"ml.machine_memory":"8588910592","ml.max_open_jobs":"512","xpack.installed":"true","ml.max_jvm_size":"2077753344","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"enable","decision":"NO","explanation":"replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"},{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[.apm-agent-configuration][0], node[2J1nm1GcRP6ofx3uYOorcA], [P], s[STARTED], a[id=PzHJSO2-Ty2gxu0_3RdXQQ]]"}]},{"node_id":"KYo2-qPbSDCfs9S59sB5Eg","node_name":"ICA-DB1","transport_address":"192.168.100.211:9300","node_attributes":{"ml.machine_memory":"8588910592","ml.max_open_jobs":"512","xpack.installed":"true","ml.max_jvm_size":"2077753344","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"enable","decision":"NO","explanation":"replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"}]},{"node_id":"TBQKi5SeQZ2rkXCflDV0LQ","node_name":"ICA-DB3","transport_address":"192.168.100.213:9300","node_attributes":{"ml.machine_memory":"8588910592","ml.max_open_jobs":"512","xpack.installed":"true","ml.max_jvm_size":"2077753344","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"enable","decision":"NO","explanation":"replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"}]},{"node_id":"eByZYAthT8O8zvMz4bufHA","node_name":"ICA-DB2","transport_address":"192.168.100.212:9300","node_attributes":{"ml.machine_memory":"8588910592","ml.max_open_jobs":"512","xpack.installed":"true","ml.max_jvm_size":"2077753344","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"enable","decision":"NO","explanation":"replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"}]},{"node_id":"s7sT_BywRuqjQZVcRKgTRA","node_name":"ICA-DB4","transport_address":"192.168.100.214:9300","node_attributes":{"ml.machine_memory":"8588910592","ml.max_open_jobs":"512","xpack.installed":"true","ml.max_jvm_size":"2077753344","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"enable","decision":"NO","explanation":"replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"}]}]}
C

mujtabah · July 31, 2022, 3:24pm

C:\Users\Administrator> curl -u "xyz:xyz" -XGET https://192.168.100.211:9200/_cat/nodes?v --ssl-no-revoke
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.100.213 28 60 0 cdfhilmrstw * ICA-DB3
192.168.100.214 49 67 0 cdfhilrstw - ICA-DB4
192.168.100.212 18 62 0 cdfhilmrstw - ICA-DB2
192.168.100.215 35 60 0 cdfhilrstw - ICA-DB5
192.168.100.211 35 61 0 cdfhilmrstw - ICA-DB1

mujtabah · July 31, 2022, 3:26pm

C:\Users\Administrator> curl -u "xyz:xyz" -XGET https://192.168.100.211:9200/_cat/thread_pool --ssl-no-revoke
ICA-DB3 analyze 0 0 0
ICA-DB3 auto_complete 0 0 0
ICA-DB3 ccr 0 0 0
ICA-DB3 fetch_shard_started 0 0 0
ICA-DB3 fetch_shard_store 0 0 0
ICA-DB3 flush 0 0 0
ICA-DB3 force_merge 0 0 0
ICA-DB3 generic 0 0 0
ICA-DB3 get 0 0 0
ICA-DB3 listener 0 0 0
ICA-DB3 management 1 0 0
ICA-DB3 ml_datafeed 0 0 0
ICA-DB3 ml_job_comms 0 0 0
ICA-DB3 ml_utility 0 0 0
ICA-DB3 refresh 0 0 0
ICA-DB3 rollup_indexing 0 0 0
ICA-DB3 search 0 0 0
ICA-DB3 search_coordination 0 0 0
ICA-DB3 search_throttled 0 0 0
ICA-DB3 searchable_snapshots_cache_fetch_async 0 0 0
ICA-DB3 searchable_snapshots_cache_prewarming 0 0 0
ICA-DB3 security-crypto 0 0 0
ICA-DB3 security-token-key 0 0 0
ICA-DB3 snapshot 0 0 0
ICA-DB3 snapshot_meta 0 0 0
ICA-DB3 system_critical_read 0 0 0
ICA-DB3 system_critical_write 0 0 0
ICA-DB3 system_read 0 0 0
ICA-DB3 system_write 0 0 0
ICA-DB3 transform_indexing 0 0 0
ICA-DB3 vector_tile_generation 0 0 0
ICA-DB3 warmer 0 0 0
ICA-DB3 watcher 0 0 0
ICA-DB3 write 0 0 0
ICA-DB4 analyze 0 0 0
ICA-DB4 auto_complete 0 0 0
ICA-DB4 ccr 0 0 0
ICA-DB4 fetch_shard_started 0 0 0
ICA-DB4 fetch_shard_store 0 0 0
ICA-DB4 flush 0 0 0
ICA-DB4 force_merge 0 0 0
ICA-DB4 generic 0 0 0
ICA-DB4 get 0 0 0
ICA-DB4 listener 0 0 0
ICA-DB4 management 1 0 0
ICA-DB4 ml_datafeed 0 0 0
ICA-DB4 ml_job_comms 0 0 0
ICA-DB4 ml_utility 0 0 0
ICA-DB4 refresh 0 0 0
ICA-DB4 rollup_indexing 0 0 0
ICA-DB4 search 0 0 0
ICA-DB4 search_coordination 0 0 0
ICA-DB4 search_throttled 0 0 0
ICA-DB4 searchable_snapshots_cache_fetch_async 0 0 0
ICA-DB4 searchable_snapshots_cache_prewarming 0 0 0
ICA-DB4 security-crypto 0 0 0
ICA-DB4 security-token-key 0 0 0
ICA-DB4 snapshot 0 0 0
ICA-DB4 snapshot_meta 0 0 0
ICA-DB4 system_critical_read 0 0 0
ICA-DB4 system_critical_write 0 0 0
ICA-DB4 system_read 0 0 0
ICA-DB4 system_write 0 0 0
ICA-DB4 transform_indexing 0 0 0
ICA-DB4 vector_tile_generation 0 0 0
ICA-DB4 warmer 0 0 0
ICA-DB4 watcher 0 0 0
ICA-DB4 write 0 0 0
ICA-DB2 analyze 0 0 0
ICA-DB2 auto_complete 0 0 0
ICA-DB2 ccr 0 0 0
ICA-DB2 fetch_shard_started 0 0 0
ICA-DB2 fetch_shard_store 0 0 0
ICA-DB2 flush 0 0 0
ICA-DB2 force_merge 0 0 0
ICA-DB2 generic 0 0 0
ICA-DB2 get 0 0 0
ICA-DB2 listener 0 0 0
ICA-DB2 management 1 0 0
ICA-DB2 ml_datafeed 0 0 0
ICA-DB2 ml_job_comms 0 0 0
ICA-DB2 ml_utility 0 0 0
ICA-DB2 refresh 0 0 0
ICA-DB2 rollup_indexing 0 0 0
ICA-DB2 search 0 0 0
ICA-DB2 search_coordination 0 0 0
ICA-DB2 search_throttled 0 0 0
ICA-DB2 searchable_snapshots_cache_fetch_async 0 0 0
ICA-DB2 searchable_snapshots_cache_prewarming 0 0 0
ICA-DB2 security-crypto 0 0 0
ICA-DB2 security-token-key 0 0 0
ICA-DB2 snapshot 0 0 0
ICA-DB2 snapshot_meta 0 0 0
ICA-DB2 system_critical_read 0 0 0
ICA-DB2 system_critical_write 0 0 0
ICA-DB2 system_read 0 0 0
ICA-DB2 system_write 0 0 0
ICA-DB2 transform_indexing 0 0 0
ICA-DB2 vector_tile_generation 0 0 0
ICA-DB2 warmer 0 0 0
ICA-DB2 watcher 0 0 0
ICA-DB2 write 0 0 0
ICA-DB5 analyze 0 0 0
ICA-DB5 auto_complete 0 0 0
ICA-DB5 ccr 0 0 0
ICA-DB5 fetch_shard_started 0 0 0
ICA-DB5 fetch_shard_store 0 0 0
ICA-DB5 flush 0 0 0
ICA-DB5 force_merge 0 0 0
ICA-DB5 generic 0 0 0
ICA-DB5 get 0 0 0
ICA-DB5 listener 0 0 0
ICA-DB5 management 1 0 0
ICA-DB5 ml_datafeed 0 0 0
ICA-DB5 ml_job_comms 0 0 0
ICA-DB5 ml_utility 0 0 0
ICA-DB5 refresh 0 0 0
ICA-DB5 rollup_indexing 0 0 0
ICA-DB5 search 0 0 0
ICA-DB5 search_coordination 0 0 0
ICA-DB5 search_throttled 0 0 0
ICA-DB5 searchable_snapshots_cache_fetch_async 0 0 0
ICA-DB5 searchable_snapshots_cache_prewarming 0 0 0
ICA-DB5 security-crypto 0 0 0
ICA-DB5 security-token-key 0 0 0
ICA-DB5 snapshot 0 0 0
ICA-DB5 snapshot_meta 0 0 0
ICA-DB5 system_critical_read 0 0 0
ICA-DB5 system_critical_write 0 0 0
ICA-DB5 system_read 0 0 0
ICA-DB5 system_write 0 0 0
ICA-DB5 transform_indexing 0 0 0
ICA-DB5 vector_tile_generation 0 0 0
ICA-DB5 warmer 0 0 0
ICA-DB5 watcher 0 0 0
ICA-DB5 write 0 0 0
ICA-DB1 analyze 0 0 0
ICA-DB1 auto_complete 0 0 0
ICA-DB1 ccr 0 0 0
ICA-DB1 fetch_shard_started 0 0 0
ICA-DB1 fetch_shard_store 0 0 0
ICA-DB1 flush 0 0 0
ICA-DB1 force_merge 0 0 0
ICA-DB1 generic 0 0 0
ICA-DB1 get 0 0 0
ICA-DB1 listener 0 0 0
ICA-DB1 management 1 0 0
ICA-DB1 ml_datafeed 0 0 0
ICA-DB1 ml_job_comms 0 0 0
ICA-DB1 ml_utility 0 0 0
ICA-DB1 refresh 0 0 0
ICA-DB1 rollup_indexing 0 0 0
ICA-DB1 search 0 0 0
ICA-DB1 search_coordination 0 0 0
ICA-DB1 search_throttled 0 0 0
ICA-DB1 searchable_snapshots_cache_fetch_async 0 0 0
ICA-DB1 searchable_snapshots_cache_prewarming 0 0 0
ICA-DB1 security-crypto 0 0 0
ICA-DB1 security-token-key 0 0 0
ICA-DB1 snapshot 0 0 0
ICA-DB1 snapshot_meta 0 0 0
ICA-DB1 system_critical_read 0 0 0
ICA-DB1 system_critical_write 0 0 0
ICA-DB1 system_read 0 0 0
ICA-DB1 system_write 0 0 0
ICA-DB1 transform_indexing 0 0 0
ICA-DB1 vector_tile_generation 0 0 0
ICA-DB1 warmer 0 0 0
ICA-DB1 watcher 0 0 0
ICA-DB1 write 0 0 0

ropc · August 1, 2022, 12:25am

@mujtabah Can you reset the cluster setting cluster.routing.allocation.enable? It is currently set to primaries , this is why your cluster is in a yellow state and may cause issues with the Kibana saved objects migration. If you have set this setting as a persistent cluster setting, you can use the below API to reset its value:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

system · August 29, 2022, 12:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.