Hi guys i have a running cluster with 5 nodes(3master and 2 data nodes). i have done rolling upgrade of elastic for all the 5 nodes from v7.10.0 to v7.16.2 and it was successfully upgraded and all the 5 nodes are in cluster and cluster health is yellow. then i tried to do standard upgrade for kibana of all 5 nodes from v7.10.0 to v7.16.0. After i done upgrade for kibana , then i run kibana batch file parallel in all 5 nodes. iam getting this error.
PS C:\kibana-7.16.2-windows-x86_64\bin> .\kibana.bat
log [15:35:04.002] [info][plugins-service] Plugin "metricsEntities" is disabled.
log [15:35:04.185] [info][server][Preboot][http] http server running at https://0.0.0.0:5601/report
log [15:35:04.307] [warning][config][deprecation] "server.cors" is deprecated and has been replaced by "server.cors.enabled"
log [15:35:04.309] [warning][config][deprecation] Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format.
log [15:35:04.311] [warning][config][deprecation] Use Kibana application privileges to grant reporting privileges. Using "xpack.reporting.roles.allow" to grant reporting privileges is deprecated. The "xpack.reporting.roles.enabled" setting will default to false in a future release.
log [15:35:04.312] [warning][config][deprecation] User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout.
log [15:35:04.313] [warning][config][deprecation] Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout.
log [15:35:04.569] [info][plugins-system][standard] Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]
log [15:35:04.611] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 7d35f6fb-daed-4ee1-adb2-fcff2bde54d5
log [15:35:04.871] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:04.921] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:04.999] [warning][encryptedSavedObjects][plugins] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.038] [warning][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.069] [warning][alerting][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [15:35:05.112] [info][plugins][ruleRegistry] Installing common resources shared between all indices
log [15:35:06.268] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox.
log [15:35:10.111] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [15:35:10.113] [info][savedobjects-service] Starting saved objects migrations
log [15:35:11.219] [info][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 26ms.
log [15:35:11.225] [info][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 6ms.
log [15:35:11.245] [info][savedobjects-service] [.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 20ms.
log [15:35:11.251] [info][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 6ms.
log [15:35:11.267] [info][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 77ms.
log [15:35:11.270] [info][savedobjects-service] [.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 19ms.
log [15:35:11.274] [info][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 7ms.
log [15:35:11.284] [info][savedobjects-service] [.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 14ms.
log [15:35:11.288] [info][savedobjects-service] [.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 14ms.
log [15:35:11.291] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 7ms.
log [15:35:11.293] [info][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 5ms.
log [15:35:11.303] [info][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 10ms.
log [15:35:11.313] [info][savedobjects-service] [.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 10ms.
log [15:35:11.320] [info][savedobjects-service] [.kibana_task_manager] Starting to process 6 documents.
log [15:35:11.321] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 29ms.
log [15:35:11.328] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 8ms.
log [15:35:11.332] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 19ms.
log [15:35:11.665] [info][savedobjects-service] [.kibana] Starting to process 48 documents.
log [15:35:11.666] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 333ms.
log [15:35:11.737] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 72ms.
log [15:37:11.403] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
log [15:37:11.405] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120075ms.
log [15:37:12.104] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
log [15:37:12.105] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120367ms.
log [15:39:13.490] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
log [15:39:13.491] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 122087ms.
log [15:39:14.490] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
log [15:39:14.491] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 122386ms.
log [15:41:17.544] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
log [15:41:17.546] [info][savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 124054ms.
log [15:41:18.841] [error][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
log [15:41:18.842] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 124351ms.