Unable to authenticate user [elastic] for REST request

wajika · April 18, 2021, 1:53am

I upgraded the ELK cluster on a rolling basis according to the documentation (7.4.2 to 7.12.0), and I only have three master/node nodes.
I rolled each es node in turn, and then reinstalled the new version of kibana. An error occurred when kibana was started.

I suspect that a certain step is missing, but it is not mentioned in the document.

Apr 18 09:39:43 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:39:43+08:00","tags":["error","elasticsearch"],"pid":7451,"message":"Request error, retrying\nGET http://192.168.10.139:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 192.168.10.139:9200"}
Apr 18 09:39:43 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:39:43+08:00","tags":["warning","plugins","licensing"],"pid":7451,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack?accept_enterprise=true\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] forREST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}
Apr 18 09:40:13 kibana kibana[7451]: {"type":"log","@timestamp":"2021-04-18T09:40:13+08:00","tags":["warning","plugins","licensing"],"pid":7451,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack?accept_enterprise=true\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] forREST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elastic] for REST request [/_xpack?accept_enterprise=true]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}

I turned off the login authentication of xpack, and I can get information from curl, but I want to know the official recommended steps.

wajika · April 18, 2021, 2:43am

Now kibana has been waiting for objects migrations, but I don’t know how long it will take.

{"type":"log","@timestamp":"2021-04-18T10:37:15+08:00","tags":["info","plugins-system"],"pid":7647,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-04-18T10:37:15+08:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7647,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-04-18T10:37:31+08:00","tags":["warning","plugins","licensing"],"pid":7647,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-04-18T10:37:37+08:00","tags":["error","savedobjects-service"],"pid":7647,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 3 out of 10 in 8 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:37:37+08:00","tags":["info","savedobjects-service"],"pid":7647,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:37:38+08:00","tags":["error","savedobjects-service"],"pid":7647,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 3 out of 10 in 8 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:37:38+08:00","tags":["info","savedobjects-service"],"pid":7647,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:37:45+08:00","tags":["warning","plugins-system"],"pid":7647,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins-service"],"pid":7679,"message":"Plugin \"osquery\" is disabled."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","config","deprecation"],"pid":7679,"message":"Setting [monitoring.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins-system"],"pid":7679,"message":"Setting up [100] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["info","plugins","taskManager"],"pid":7679,"message":"TaskManager is identified by the Kibana UUID: 2551ee7f-e607-40b7-b253-6c9f14c49c73"}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","security","config"],"pid":7679,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","security","config"],"pid":7679,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","reporting","config"],"pid":7679,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","reporting","config"],"pid":7679,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.5.1804 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":7679,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","fleet"],"pid":7679,"message":"Fleet APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","actions","actions"],"pid":7679,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:43+08:00","tags":["warning","plugins","alerts","plugins","alerting"],"pid":7679,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","plugins","monitoring","monitoring"],"pid":7679,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] INIT -> SET_SOURCE_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] INIT -> SET_SOURCE_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK"}
{"type":"log","@timestamp":"2021-04-18T10:38:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["error","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 out of 10 in 2 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["error","savedobjects-service"],"pid":7679,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 1 out of 10 in 2 seconds."}
{"type":"log","@timestamp":"2021-04-18T10:40:44+08:00","tags":["info","savedobjects-service"],"pid":7679,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET"}

wajika · April 18, 2021, 4:41am

github.com/elastic/kibana

7.12.0 upgrade migrations fail with timeout_exception or receive_timeout_transport_exception

opened 03:24PM - 24 Mar 21 UTC

closed 08:24AM - 02 Apr 21 UTC

rudolf

bug Team:Core v7.12.0 project:ResilientSavedObjectMigrations

v7.12.0 migrations can fail if the `.kibana` index has a large number of saved o…bjects or the Elasticsearch cluster is under heavy load. This will cause errors logs like: > [.kibana] [receive_timeout_transport_exception]: [instance-0000000002][10.42.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] > [.kibana] [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] The root cause is a bug in the v2 migrations that means Kibana will only wait for 60s for some of the steps. When a cluster is busy or there are a lot of `fleet-agent-events` `_reindex` or `_update_by_query` tasks can take longer than 60s leading to a timeout. # Workaround for large fleet-agent-events If you've used fleet and have a large number of fleet-agent-events follow this workaround 1. First establish that there are large number of fleet-agent-events: ``` POST .kibana/_search?filter_path=aggregations { "aggs": { "saved_object_type": { "terms": {"field": "type"} } } } ``` 2. Shutdown Kibana 3. Wait for any reindex tasks started by Kibana to complete. The output of the following request shouldn't show any reindex tasks for the `.kibana*` indices: ``` GET _tasks?actions=*update/byquery,reindex&group_by=parents&detailed # Cancel any tasks where the "description" field shows it's operating on a `.kibana*` index # e.g. `"description": "update-by-query [.kibana_7.12.0_001]" POST _tasks/<task id from above>/_cancel # Verify that all tasks have been cancelled: GET _tasks?actions=*update/byquery,reindex&group_by=parents&detailed ``` 3. Delete fleet-agent-events ``` POST .kibana,.kibana_7.12.0_*/_delete_by_query?conflicts=proceed&wait_for_completion=false { "query": { "bool": { "must": { "term": { "type": "fleet-agent-events" } } } } } ``` Check that the task returned from the above operation has completed before proceeding ``` GET _tasks/<id> ``` 4. Restart Kibana PR https://github.com/elastic/kibana/pull/95305

According to the information I found, I got some information, but my data value is very small.

curl -H "Content-type: application/json" -XGET 'http://elastic:elastic@192.168.10.140:9200/.kibana/_search?filter_path=aggregations' -d '
> {
>   "aggs": {
>     "saved_object_type": {
>       "terms": {"field": "type"}
>     }
>   }
> }'

{
	"aggregations": {
		"saved_object_type": {
			"doc_count_error_upper_bound": 0,
			"sum_other_doc_count": 17,
			"buckets": [{
				"key": "application_usage_transactional",
				"doc_count": 4938
			}, {
				"key": "visualization",
				"doc_count": 677
			}, {
				"key": "ui-metric",
				"doc_count": 193
			}, {
				"key": "dashboard",
				"doc_count": 108
			}, {
				"key": "search",
				"doc_count": 94
			}, {
				"key": "index-pattern",
				"doc_count": 52
			}, {
				"key": "config",
				"doc_count": 20
			}, {
				"key": "application_usage_totals",
				"doc_count": 18
			}, {
				"key": "map",
				"doc_count": 11
			}, {
				"key": "space",
				"doc_count": 10
			}]
		}
	}
}

curl -H "Content-type: application/json" -XGET 'http://elastic:elastic@192.168.10.140:9200/.kibana_task_manager/_search?filter_path=aggregations' -d '
> {
>   "aggs": {
>     "saved_object_type": {
>       "terms": {"field": "type"}
>     }
>   }
> }'

{
	"aggregations": {
		"saved_object_type": {
			"doc_count_error_upper_bound": 0,
			"sum_other_doc_count": 0,
			"buckets": [{
				"key": "task",
				"doc_count": 6
			}]
		}
	}
}

wajika · April 19, 2021, 12:58am

Finally, I delete all kibana indexes and restart kibana. After that, the cluster returns to normal.

But I doubt that this operation will lose some data. I hope the elastic team can give a best practice.

rashmi · April 19, 2021, 6:07pm

it's possible something is going wrong during the migration if there are corrupted saved objects (this can happen for a bunch of reasons). Please stop Kibana, delete the .kibana_2 index again, restart Kibana and check the logs for error messages. It's likely there are some related to "migration" - please copy/paste them here. Upgrade migrations | Kibana Guide [7.12] | Elastic To prevent migrations from failing again make sure you read the section on "preventing migration failures" before you retry the migration.
Hope it helps,
Thanks
Rashmi

wajika · April 20, 2021, 12:22am

@rashmi

Hello, rashmi.
I deleted .kibana and .kibana_task_manager, and now I can successfully access the kibana. but what I am really worried about is whether I will lose any data? such as "visualization and dashboard".

In addition, I am more concerned about why there are data conflicts in the upgrade from 7.X to 7.X? This is not a cross-version upgrade.
I thought the version of the update is very safe thing.

rashmi · May 3, 2021, 5:49am

@rudolf can you please shed more light here ?
Thanks
Rashmi

system · May 31, 2021, 5:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.