ERROR: Failed to set password for user [apm_system] yet again

Im running elasticsearch 7.6.2 Basic license

3 master nodes

cluster:
  name: dev
node:
  master: true
  data: false
  ingest: false

network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true

3 data nodes

cluster.name: dev
node:
  master: false
  data: true
  ingest: true

network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true

If I set, the cluster comes up fine

xpack.security.enabled: false

When I set, the clustering seems fine.

xpack.security.enabled: true

But when I run

bin/elasticsearch-setup-passwords auto -b --verbose

Cluster is green, but

Trying user password change call http://10.1.46.122:9200/_security/user/apm_system/_password?pretty

Connection failure to: http://10.1.46.122:9200/_security/user/apm_system/_password?pretty failed: Read timed out

java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:284)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:310)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:937)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:245)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:285)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:344)
        at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
        at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:119)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changeUserPassword(SetupPasswordTool.java:462)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changePasswords(SetupPasswordTool.java:523)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:143)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:107)


ERROR: Failed to set password for user [apm_system].

master log

{"type": "server", "timestamp": "2020-05-21T21:36:30,274Z", "level": "INFO", "component": "o.e.x.s.s.SecurityIndexManager", "cluster.name": "dev", "node.name": "es-master-0", "message": "security index does not exist. Creating [.security-7] with alias [.security]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:36:30,447Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "dev", "node.name": "es-master-0", "message": "[.security-7] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:36:30,458Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "dev", "node.name": "es-master-0", "message": "Cluster health status changed from [YELLOW] to [RED] (reason: [index [.security-7] created]).", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:38:00,568Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "dev", "node.name": "es-master-0", "message": "path: /_security/user/apm_system/_password, params: {pretty=, username=apm_system}", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: [.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update",

I read a bunch of other similar issues on this forum about having data nodes available which I do have.
The only way I made it work was to make the master include data role as well.

3 master nodes

node:
  master: true
  data: true

Why can I not have separate master and data only nodes ?

Problem solved. Indeed there was not data nodes as the data node configuration was pointing to the master configmap.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.