Unable to authenticate user [elastic] for REST request

I upgraded the ELK cluster on a rolling basis according to the documentation (7.4.2 to 7.12.0), and I only have three master/node nodes.
I rolled each es node in turn, and then reinstalled the new version of kibana. An error occurred when kibana was started.

I suspect that a certain step is missing, but it is not mentioned in the document.

Apr 18 09:39:43 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:39:43+08:00","tags":["error","elasticsearch"],"pid":7451,"message":"Request error, retrying\nGET http://192.168.10.139:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 192.168.10.139:9200"}
Apr 18 09:39:43 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:39:43+08:00","tags":["warning","plugins","licensing"],"pid":7451,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack?accept_enterprise=true\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] forREST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}
Apr 18 09:40:13 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:40:13+08:00","tags":["warning","plugins","licensing"],"pid":7451,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack?accept_enterprise=true\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] forREST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}

I turned off the login authentication of xpack, and I can get information from curl, but I want to know the official recommended steps.

Now kibana has been waiting for objects migrations, but I don’t know how long it will take.

{"type":"log","@timestamp":"2021-04-18T10:37:15+08:00","tags":["info","plugins-system"],"pid":7647,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-04-18T10:37:15+08:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7647,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-04-18T10:37:31+08:00","tags":["warning","plugins","licensing"],"pid":7647,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-04-18T10:37:37+08:00","tags":["error","savedobjects-service"],"pid":7647,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 3 out of 10 in 8 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:37:37+08:00","tags":["info","savedobjects-service"],"pid":7647,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:37:38+08:00","tags":["error","savedobjects-service"],"pid":7647,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 3 out of 10 in 8 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:37:38+08:00","tags":["info","savedobjects-service"],"pid":7647,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:37:45+08:00","tags":["warning","plugins-system"],"pid":7647,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins-service"],"pid":7679,"message":"Plugin \"osquery\" is disabled."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Setting [monitoring.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins-system"],"pid":7679,"message":"Setting up [100] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins","taskManager"],"pid":7679,"message":"TaskManager is identified by the Kibana UUID: 2551ee7f-e607-40b7-b253-6c9f14c49c73"}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","security","config"],"pid":7679,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","security","config"],"pid":7679,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","reporting","config"],"pid":7679,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","reporting","config"],"pid":7679,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.5.1804 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":7679,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","fleet"],"pid":7679,"message":"Fleet APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","actions","actions"],"pid":7679,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","alerts","plugins","alerting"],"pid":7679,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","plugins","monitoring","monitoring"],"pid":7679,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] INIT -> SET_SOURCE_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] INIT -> SET_SOURCE_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["error","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 out of 10 in 2 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["error","savedobjects-service"],"pid":7679,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 1 out of 10 in 2 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}

According to the information I found, I got some information, but my data value is very small.

curl -H "Content-type: application/json" -XGET 'http://elastic:elastic@192.168.10.140:9200/.kibana/_search?filter_path=aggregations' -d '
> {
>   "aggs": {
>     "saved_object_type": {
>       "terms": {"field": "type"}
>     }
>   }
> }'
{
	"aggregations": {
		"saved_object_type": {
			"doc_count_error_upper_bound": 0,
			"sum_other_doc_count": 17,
			"buckets": [{
				"key": "application_usage_transactional",
				"doc_count": 4938
			}, {
				"key": "visualization",
				"doc_count": 677
			}, {
				"key": "ui-metric",
				"doc_count": 193
			}, {
				"key": "dashboard",
				"doc_count": 108
			}, {
				"key": "search",
				"doc_count": 94
			}, {
				"key": "index-pattern",
				"doc_count": 52
			}, {
				"key": "config",
				"doc_count": 20
			}, {
				"key": "application_usage_totals",
				"doc_count": 18
			}, {
				"key": "map",
				"doc_count": 11
			}, {
				"key": "space",
				"doc_count": 10
			}]
		}
	}
}
curl -H "Content-type: application/json" -XGET 'http://elastic:elastic@192.168.10.140:9200/.kibana_task_manager/_search?filter_path=aggregations' -d '
> {
>   "aggs": {
>     "saved_object_type": {
>       "terms": {"field": "type"}
>     }
>   }
> }'
{
	"aggregations": {
		"saved_object_type": {
			"doc_count_error_upper_bound": 0,
			"sum_other_doc_count": 0,
			"buckets": [{
				"key": "task",
				"doc_count": 6
			}]
		}
	}
}

Finally, I delete all kibana indexes and restart kibana. After that, the cluster returns to normal.

But I doubt that this operation will lose some data. I hope the elastic team can give a best practice.

it's possible something is going wrong during the migration if there are corrupted saved objects (this can happen for a bunch of reasons). Please stop Kibana, delete the .kibana_2 index again, restart Kibana and check the logs for error messages. It's likely there are some related to "migration" - please copy/paste them here. Upgrade migrations | Kibana Guide [7.12] | Elastic To prevent migrations from failing again make sure you read the section on "preventing migration failures" before you retry the migration.
Hope it helps,
Thanks
Rashmi

@rashmi

Hello, rashmi.
I deleted .kibana and .kibana_task_manager, and now I can successfully access the kibana. but what I am really worried about is whether I will lose any data? such as "visualization and dashboard".

In addition, I am more concerned about why there are data conflicts in the upgrade from 7.X to 7.X? This is not a cross-version upgrade.
I thought the version of the update is very safe thing.

@rudolf can you please shed more light here ?
Thanks
Rashmi