# Authentication is not working after it got enabled

**URL:** <https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [December 4, 2020, 5:36am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611 "2020-12-04T05:36:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![luok0](https://avatars.discourse-cdn.com/v4/letter/l/5fc32e/32.png) [@luok0](https://discuss.elastic.co/u/luok0)\
**Post date:** [December 4, 2020, 5:36am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/1 "2020-12-04T05:36:50Z")

</div>

I enabled the authentication in elasticsearch.yml as below:

```auto
# http.enabled: false
 xpack.security.enabled: true

```

It worked well with below:

```auto
curl -u elastic:xxoo 'http://localhost:9200/?pretty'
{
  "name" : "kkj6-5Y",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "G6ANVMwSRwCOpPHDW7nFJQ",
  "version" : {
    "number" : "6.7.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "56c6e48",
    "build_date" : "2019-04-29T09:05:50.290371Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

```

However, it's not working with kibana and logstash.  
Error in kibana:

```auto
failed to authenticate user [kibana]

{"type":"log","@timestamp":"2020-12-03T15:13:36Z","tags":["error","task_manager"],"pid":25751,"message":"Failed to poll for work: [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"terms\\\":{\\\"task.taskType\\\":[\\\"maps_telemetry\\\",\\\"vis_telemetry\\\"]}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lte\\\":3}}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}},{\\\"range\\\":{\\\"kibana.apiVersion\\\":{\\\"lte\\\":1}}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"task.runAt\\\":{\\\"order\\\":\\\"asc\\\"}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

```

Error in logstash:

```auto
[ERROR] 2020-12-03 23:15:32.783 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>401, :url=>"http://localhost:9200/_bulk"}
[ERROR] 2020-12-03 23:15:34.803 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>401, :url=>"http://localhost:9200/_bulk"}
[ERROR] 2020-12-03 23:15:38.812 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>401, :url=>"http://localhost:9200/_bulk"}

```

Kibana.yml as below:

```auto
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "elastic"
elasticsearch.password: "xxoo"

```

logstash.yml as below:

```auto
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxoo

```

Could anyone help me out with it?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 4, 2020, 6:16am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/2 "2020-12-04T06:16:35Z")

</div>

Could you share the full elasticsearch logs when you restart?

If it's a Kibana configuration issue, I'd suggest to move your question to #elastic-stack:kibana

---

<div class="post-metadata">

**Author:** ![luok0](https://avatars.discourse-cdn.com/v4/letter/l/5fc32e/32.png) [@luok0](https://discuss.elastic.co/u/luok0)\
**Post date:** [December 4, 2020, 6:40am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/3 "2020-12-04T06:40:32Z")

</div>

Kindly check the log as attached. Thanks.

```auto
[2020-12-03T23:17:57,469][INFO][o.e.c.s.ClusterSettings] [kkj6-5Y] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-12-03T23:17:59,334][INFO][o.e.l.LicenseService] [kkj6-5Y] license [54503db5-9bef-49a4-b03a-ffed4f23bfc6] mode [trial] - valid
[2020-12-03T23:17:59,341][INFO][o.e.g.GatewayService] [kkj6-5Y] recovered [322] indices into cluster_state
[2020-12-03T23:17:59,929][INFO][o.e.x.w.WatcherService] [kkj6-5Y] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2020-12-03T23:18:20,415][INFO][o.e.c.r.a.AllocationService] [kkj6-5Y] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][2], [test][1], [test][3]] ...]).
[2020-12-04T00:00:05,810][INFO][o.e.c.m.MetaDataCreateIndexService] [kkj6-5Y] [heartbeat-6.8.1-2020.12.04] creating index, cause [auto(bulk api)], templates [heartbeat-6.8.1], shards [1]/[1], mappings [doc]
[2020-12-04T01:00:00,007][INFO][o.e.x.m.e.l.LocalExporter] [kkj6-5Y] cleaning up [2] old indices
[2020-12-04T01:00:00,012][INFO][o.e.c.m.MetaDataDeleteIndexService] [kkj6-5Y] [.monitoring-kibana-6-2020.11.27/oAYHGfrERS-WprIXraiIfQ] deleting index
[2020-12-04T01:00:00,012][INFO][o.e.c.m.MetaDataDeleteIndexService] [kkj6-5Y] [.monitoring-es-6-2020.11.27/BaaIMNyCShiok7MVEZUs6g] deleting index
[2020-12-04T01:25:35,267][INFO][o.e.n.Node] [kkj6-5Y] stopping ...
[2020-12-04T01:25:35,277][INFO][o.e.x.w.WatcherService] [kkj6-5Y] stopping watch service, reason [shutdown initiated]
[2020-12-04T01:25:35,479][INFO][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/43916] [Main.cc@148] Ml controller exiting
[2020-12-04T01:25:35,480][INFO][o.e.x.m.p.NativeController] [kkj6-5Y] Native controller process has stopped - no new native processes can be started
[2020-12-04T01:25:35,906][INFO][o.e.n.Node] [kkj6-5Y] stopped
[2020-12-04T01:25:35,906][INFO][o.e.n.Node] [kkj6-5Y] closing ...
[2020-12-04T01:25:35,930][INFO][o.e.n.Node] [kkj6-5Y] closed
[2020-12-04T01:26:34,036][INFO][o.e.e.NodeEnvironment] [kkj6-5Y] using [1] data paths, mounts [[/ (/dev/mapper/rootvg-lv_root)]], net usable_space [45.8gb], net total_space [97.4gb], types [xfs]
[2020-12-04T01:26:34,040][INFO][o.e.e.NodeEnvironment] [kkj6-5Y] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-12-04T01:26:34,827][INFO][o.e.n.Node] [kkj6-5Y] node name derived from node ID [kkj6-5Y0QGuU64lYWhZ5qg]; set [node.name] to override
[2020-12-04T01:26:34,827][INFO][o.e.n.Node] [kkj6-5Y] version[6.7.2], pid[52579], build[default/tar/56c6e48/2019-04-29T09:05:50.290371Z], OS[Linux/3.10.0-1160.2.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_261/25.261-b12]
[2020-12-04T01:26:34,828][INFO][o.e.n.Node] [kkj6-5Y] JVM arguments [-Xms1g, -Xmx1g, -Djavax.net.ssl.trustStore=/etc/elasticsearch/ssl/cacerts/cacerts.jks, -Djavax.net.ssl.trustStorePassword=changeit, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-968002765459257682, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Djavax.net.ssl.trustStore=/etc/elasticsearch/ssl/cacerts/cacerts.jks, -Djavax.net.ssl.trustStorePassword=changeit, -Des.allow_insecure_settings=true, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [aggs-matrix-stats]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [analysis-common]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [ingest-common]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [ingest-geoip]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [ingest-user-agent]
[2020-12-04T01:26:37,146][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [lang-expression]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [lang-mustache]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [lang-painless]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [mapper-extras]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [parent-join]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [percolator]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [rank-eval]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [reindex]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [repository-url]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [transport-netty4]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [tribe]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-ccr]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-core]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-deprecation]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-graph]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-ilm]
[2020-12-04T01:26:37,147][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-logstash]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-ml]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-monitoring]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-rollup]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-security]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-sql]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-upgrade]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded module [x-pack-watcher]
[2020-12-04T01:26:37,148][INFO][o.e.p.PluginsService] [kkj6-5Y] loaded plugin [repository-s3]
[2020-12-04T01:26:41,483][INFO][o.e.x.s.a.s.FileRolesStore] [kkj6-5Y] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-12-04T01:26:42,229][INFO][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/52608] [Main.cc@109] controller (64 bit): Version 6.7.2 (Build f688467e0867ca) Copyright (c) 2019 Elasticsearch BV
[2020-12-04T01:26:42,863][DEBUG][o.e.a.ActionModule] [kkj6-5Y] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-12-04T01:26:43,908][INFO][o.e.d.DiscoveryModule] [kkj6-5Y] using discovery type [single-node] and host providers [settings]
[2020-12-04T01:26:44,830][INFO][o.e.n.Node] [kkj6-5Y] initialized
[2020-12-04T01:26:44,830][INFO][o.e.n.Node] [kkj6-5Y] starting ...
[2020-12-04T01:26:45,006][INFO][o.e.t.TransportService] [kkj6-5Y] publish_address {10.210.87.190:9300}, bound_addresses {0.0.0.0:9300}
[2020-12-04T01:26:45,325][INFO][o.e.h.n.Netty4HttpServerTransport] [kkj6-5Y] publish_address {10.210.87.190:9200}, bound_addresses {0.0.0.0:9200}
[2020-12-04T01:26:45,325][INFO][o.e.n.Node] [kkj6-5Y] started
[2020-12-04T01:26:47,326][INFO][o.e.c.s.ClusterSettings] [kkj6-5Y] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-12-04T01:26:49,170][INFO][o.e.l.LicenseService] [kkj6-5Y] license [54503db5-9bef-49a4-b03a-ffed4f23bfc6] mode [trial] - valid
[2020-12-04T01:26:49,180][INFO][o.e.g.GatewayService] [kkj6-5Y] recovered [321] indices into cluster_state
[2020-12-04T01:27:10,477][INFO][o.e.c.r.a.AllocationService] [kkj6-5Y] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][2]] ...]).
[2020-12-04T01:28:02,339][INFO][o.e.x.s.a.AuthenticationService] [kkj6-5Y] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2020-12-04T01:31:36,389][INFO][o.e.n.Node] [kkj6-5Y] stopping ...
[2020-12-04T01:31:36,402][INFO][o.e.x.w.WatcherService] [kkj6-5Y] stopping watch service, reason [shutdown initiated]
[2020-12-04T01:31:36,427][INFO][o.e.x.m.p.l.CppLogMessageHandler] [kkj6-5Y] [controller/52608] [Main.cc@148] Ml controller exiting
[2020-12-04T01:31:36,428][INFO][o.e.x.m.p.NativeController] [kkj6-5Y] Native controller process has stopped - no new native processes can be started
[2020-12-04T01:31:36,851][INFO][o.e.n.Node] [kkj6-5Y] stopped
[2020-12-04T01:31:36,852][INFO][o.e.n.Node] [kkj6-5Y] closing ...
[2020-12-04T01:31:36,894][INFO][o.e.n.Node] [kkj6-5Y] closed

```

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [December 4, 2020, 7:25am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/4 "2020-12-04T07:25:21Z")

</div>

How did you set the password for the `elastic` user after you enabled security ? Did you run `elasticsearch-setup-passwords` CLI ? Did you set the passwords for the `kibana_admin` user and `logstash_system` user?

In your kibana.yml, you are using the `elastic` user, but you should be using the `kibana_admin` user instead.

In your logstash.yml file you have the username commented out.

---

<div class="post-metadata">

**Author:** ![luok0](https://avatars.discourse-cdn.com/v4/letter/l/5fc32e/32.png) [@luok0](https://discuss.elastic.co/u/luok0)\
**Post date:** [December 4, 2020, 7:40am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/5 "2020-12-04T07:40:33Z")

</div>

Yes, I ran like below, and I can see 'Enter password for [kibana]', which username should be correct in kibana.yml, **kibana** or **kibana\_admin**?

```auto
/elasticsearch/bin/elasticsearch-setup-passwords interactive

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

```

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [December 10, 2020, 6:11am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/6 "2020-12-10T06:11:35Z")

</div>

It was called `kibana` back in 6.7.2, it's now called `kibana_admin` . Any reason in particular why you are using such an old version ( 6.7.2 )? I urge you to upgrade and use the latest available (7.10 ) if nothing else, security is available in the basic license now. In 6.7.2, you'd need a paid license for the security features

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [December 10, 2020, 7:20am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/7 "2020-12-10T07:20:15Z")

</div>

> [@ikakavas](#):
>
> it's now called ~~`kibana_admin`~~

The user that should be used for Kibana to connect to Elasticsearch is (since v7.8) called `kibana_system`.  
(`kibana_admin` is the role that replaced `kibana_user`).

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [December 10, 2020, 9:53am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/8 "2020-12-10T09:53:48Z")

</div>

Thanks, mixed these up !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 7, 2021, 9:53am UTC](https://discuss.elastic.co/t/authentication-is-not-working-after-it-got-enabled/257611/9 "2021-01-07T09:53:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.