Authentication is not working after it got enabled

luok0 · December 4, 2020, 5:36am

I enabled the authentication in elasticsearch.yml as below:

# http.enabled: false
 xpack.security.enabled: true

It worked well with below:

curl -u elastic:xxoo 'http://localhost:9200/?pretty'
{
  "name" : "kkj6-5Y",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "G6ANVMwSRwCOpPHDW7nFJQ",
  "version" : {
    "number" : "6.7.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "56c6e48",
    "build_date" : "2019-04-29T09:05:50.290371Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

However, it's not working with kibana and logstash.
Error in kibana:

failed to authenticate user  [kibana]

{"type":"log","@timestamp":"2020-12-03T15:13:36Z","tags":["error","task_manager"],"pid":25751,"message":"Failed to poll for work: [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"terms\\\":{\\\"task.taskType\\\":[\\\"maps_telemetry\\\",\\\"vis_telemetry\\\"]}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lte\\\":3}}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}},{\\\"range\\\":{\\\"kibana.apiVersion\\\":{\\\"lte\\\":1}}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"task.runAt\\\":{\\\"order\\\":\\\"asc\\\"}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

Error in logstash:

[ERROR] 2020-12-03 23:15:32.783 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>401, :url=>"http://localhost:9200/_bulk"}
[ERROR] 2020-12-03 23:15:34.803 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>401, :url=>"http://localhost:9200/_bulk"}
[ERROR] 2020-12-03 23:15:38.812 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>401, :url=>"http://localhost:9200/_bulk"}

Kibana.yml as below:

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "elastic"
elasticsearch.password: "xxoo"

logstash.yml as below:

# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxoo

Could anyone help me out with it?

dadoonet · December 4, 2020, 6:16am

Could you share the full elasticsearch logs when you restart?

If it's a Kibana configuration issue, I'd suggest to move your question to #elastic-stack:kibana

luok0 · December 4, 2020, 6:40am

Kindly check the log as attached. Thanks.

[2020-12-03T23:17:57,469][INFO ][o.e.c.s.ClusterSettings  ] [kkj6-5Y] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-12-03T23:17:59,334][INFO ][o.e.l.LicenseService     ] [kkj6-5Y] license [54503db5-9bef-49a4-b03a-ffed4f23bfc6] mode [trial] - valid
[2020-12-03T23:17:59,341][INFO ][o.e.g.GatewayService     ] [kkj6-5Y] recovered [322] indices into cluster_state
[2020-12-03T23:17:59,929][INFO ][o.e.x.w.WatcherService   ] [kkj6-5Y] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2020-12-03T23:18:20,415][INFO ][o.e.c.r.a.AllocationService] [kkj6-5Y] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][2], [test][1], [test][3]] ...]).
[2020-12-04T00:00:05,810][INFO ][o.e.c.m.MetaDataCreateIndexService] [kkj6-5Y] [heartbeat-6.8.1-2020.12.04] creating index, cause [auto(bulk api)], templates [heartbeat-6.8.1], shards [1]/[1], mappings [doc]
[2020-12-04T01:00:00,007][INFO ][o.e.x.m.e.l.LocalExporter] [kkj6-5Y] cleaning up [2] old indices
[2020-12-04T01:00:00,012][INFO ][o.e.c.m.MetaDataDeleteIndexService] [kkj6-5Y] [.monitoring-kibana-6-2020.11.27/oAYHGfrERS-WprIXraiIfQ] deleting index
[2020-12-04T01:00:00,012][INFO ][o.e.c.m.MetaDataDeleteIndexService] [kkj6-5Y] [.monitoring-es-6-2020.11.27/BaaIMNyCShiok7MVEZUs6g] deleting index
[2020-12-04T01:25:35,267][INFO ][o.e.n.Node               ] [kkj6-5Y] stopping ...
[2020-12-04T01:25:35,277][INFO ][o.e.x.w.WatcherService   ] [kkj6-5Y] stopping watch service, reason [shutdown initiated]
[2020-12-04T01:25:35,479][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/43916] [Main.cc@148] Ml controller exiting
[2020-12-04T01:25:35,480][INFO ][o.e.x.m.p.NativeController] [kkj6-5Y] Native controller process has stopped - no new native processes can be started
[2020-12-04T01:25:35,906][INFO ][o.e.n.Node               ] [kkj6-5Y] stopped
[2020-12-04T01:25:35,906][INFO ][o.e.n.Node               ] [kkj6-5Y] closing ...
[2020-12-04T01:25:35,930][INFO ][o.e.n.Node               ] [kkj6-5Y] closed
[2020-12-04T01:26:34,036][INFO ][o.e.e.NodeEnvironment    ] [kkj6-5Y] using [1] data paths, mounts [[/ (/dev/mapper/rootvg-lv_root)]], net usable_space [45.8gb], net total_space [97.4gb], types [xfs]
[2020-12-04T01:26:34,040][INFO ][o.e.e.NodeEnvironment    ] [kkj6-5Y] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-12-04T01:26:34,827][INFO ][o.e.n.Node               ] [kkj6-5Y] node name derived from node ID [kkj6-5Y0QGuU64lYWhZ5qg]; set [node.name] to override
[2020-12-04T01:26:34,827][INFO ][o.e.n.Node               ] [kkj6-5Y] version[6.7.2], pid[52579], build[default/tar/56c6e48/2019-04-29T09:05:50.290371Z], OS[Linux/3.10.0-1160.2.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_261/25.261-b12]
[2020-12-04T01:26:34,828][INFO ][o.e.n.Node               ] [kkj6-5Y] JVM arguments [-Xms1g, -Xmx1g, -Djavax.net.ssl.trustStore=/etc/elasticsearch/ssl/cacerts/cacerts.jks, -Djavax.net.ssl.trustStorePassword=changeit, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-968002765459257682, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Djavax.net.ssl.trustStore=/etc/elasticsearch/ssl/cacerts/cacerts.jks, -Djavax.net.ssl.trustStorePassword=changeit, -Des.allow_insecure_settings=true, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [aggs-matrix-stats]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [analysis-common]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [ingest-common]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [ingest-geoip]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [ingest-user-agent]
[2020-12-04T01:26:37,146][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [lang-expression]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [lang-mustache]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [lang-painless]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [mapper-extras]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [parent-join]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [percolator]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [rank-eval]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [reindex]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [repository-url]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [transport-netty4]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [tribe]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-ccr]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-core]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-deprecation]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-graph]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-ilm]
[2020-12-04T01:26:37,147][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-logstash]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-ml]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-monitoring]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-rollup]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-security]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-sql]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-upgrade]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded module [x-pack-watcher]
[2020-12-04T01:26:37,148][INFO ][o.e.p.PluginsService     ] [kkj6-5Y] loaded plugin [repository-s3]
[2020-12-04T01:26:41,483][INFO ][o.e.x.s.a.s.FileRolesStore] [kkj6-5Y] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-12-04T01:26:42,229][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/52608] [Main.cc@109] controller (64 bit): Version 6.7.2 (Build f688467e0867ca) Copyright (c) 2019 Elasticsearch BV
[2020-12-04T01:26:42,863][DEBUG][o.e.a.ActionModule       ] [kkj6-5Y] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-12-04T01:26:43,908][INFO ][o.e.d.DiscoveryModule    ] [kkj6-5Y] using discovery type [single-node] and host providers [settings]
[2020-12-04T01:26:44,830][INFO ][o.e.n.Node               ] [kkj6-5Y] initialized
[2020-12-04T01:26:44,830][INFO ][o.e.n.Node               ] [kkj6-5Y] starting ...
[2020-12-04T01:26:45,006][INFO ][o.e.t.TransportService   ] [kkj6-5Y] publish_address {10.210.87.190:9300}, bound_addresses {0.0.0.0:9300}
[2020-12-04T01:26:45,325][INFO ][o.e.h.n.Netty4HttpServerTransport] [kkj6-5Y] publish_address {10.210.87.190:9200}, bound_addresses {0.0.0.0:9200}
[2020-12-04T01:26:45,325][INFO ][o.e.n.Node               ] [kkj6-5Y] started
[2020-12-04T01:26:47,326][INFO ][o.e.c.s.ClusterSettings  ] [kkj6-5Y] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-12-04T01:26:49,170][INFO ][o.e.l.LicenseService     ] [kkj6-5Y] license [54503db5-9bef-49a4-b03a-ffed4f23bfc6] mode [trial] - valid
[2020-12-04T01:26:49,180][INFO ][o.e.g.GatewayService     ] [kkj6-5Y] recovered [321] indices into cluster_state
[2020-12-04T01:27:10,477][INFO ][o.e.c.r.a.AllocationService] [kkj6-5Y] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][2]] ...]).
[2020-12-04T01:28:02,339][INFO ][o.e.x.s.a.AuthenticationService] [kkj6-5Y] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2020-12-04T01:31:36,389][INFO ][o.e.n.Node               ] [kkj6-5Y] stopping ...
[2020-12-04T01:31:36,402][INFO ][o.e.x.w.WatcherService   ] [kkj6-5Y] stopping watch service, reason [shutdown initiated]
[2020-12-04T01:31:36,427][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/52608] [Main.cc@148] Ml controller exiting
[2020-12-04T01:31:36,428][INFO ][o.e.x.m.p.NativeController] [kkj6-5Y] Native controller process has stopped - no new native processes can be started
[2020-12-04T01:31:36,851][INFO ][o.e.n.Node               ] [kkj6-5Y] stopped
[2020-12-04T01:31:36,852][INFO ][o.e.n.Node               ] [kkj6-5Y] closing ...
[2020-12-04T01:31:36,894][INFO ][o.e.n.Node               ] [kkj6-5Y] closed

ikakavas · December 4, 2020, 7:25am

How did you set the password for the elastic user after you enabled security ? Did you run elasticsearch-setup-passwords CLI ? Did you set the passwords for the kibana_admin user and logstash_system user?

In your kibana.yml, you are using the elastic user, but you should be using the kibana_admin user instead.

In your logstash.yml file you have the username commented out.

luok0 · December 4, 2020, 7:40am

Yes, I ran like below, and I can see 'Enter password for [kibana]', which username should be correct in kibana.yml, kibana or kibana_admin?

/elasticsearch/bin/elasticsearch-setup-passwords interactive

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

ikakavas · December 10, 2020, 6:11am

It was called kibana back in 6.7.2, it's now called kibana_admin . Any reason in particular why you are using such an old version ( 6.7.2 )? I urge you to upgrade and use the latest available (7.10 ) if nothing else, security is available in the basic license now. In 6.7.2, you'd need a paid license for the security features

TimV · December 10, 2020, 7:20am

The user that should be used for Kibana to connect to Elasticsearch is (since v7.8) called kibana_system.
(kibana_admin is the role that replaced kibana_user).

ikakavas · December 10, 2020, 9:53am

Thanks, mixed these up !

system · January 7, 2021, 9:53am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problems enabling authentication to elasticsearch Kibana elastic-stack-security	6	3464	July 13, 2020
If I enable xpack , Elasticsearch stops working and my license won't work Elasticsearch elastic-stack-security	10	2480	October 22, 2020
Strange authentication error Kibana elastic-stack-security	3	1031	April 20, 2023
Cannot authenticate with kibana user to the stack after I enabled security 7.7.1 Kibana elastic-stack-security	5	1138	December 7, 2020
Kibana login does not work Kibana	5	5718	July 29, 2019

Authentication is not working after it got enabled

Related topics