ERROR: Failed to set password for user [apm_system] yet again

Elastic Stack Elasticsearch
1

Im running elasticsearch 7.6.2 Basic license

3 master nodes

cluster:
  name: dev
node:
  master: true
  data: false
  ingest: false

network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true

3 data nodes

cluster.name: dev
node:
  master: false
  data: true
  ingest: true

network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true

If I set, the cluster comes up fine

xpack.security.enabled: false

When I set, the clustering seems fine.

xpack.security.enabled: true

But when I run

bin/elasticsearch-setup-passwords auto -b --verbose

Cluster is green, but

Trying user password change call http://10.1.46.122:9200/_security/user/apm_system/_password?pretty

Connection failure to: http://10.1.46.122:9200/_security/user/apm_system/_password?pretty failed: Read timed out

java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:284)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:310)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:937)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:245)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:285)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:344)
        at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
        at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:119)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changeUserPassword(SetupPasswordTool.java:462)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changePasswords(SetupPasswordTool.java:523)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:143)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:107)


ERROR: Failed to set password for user [apm_system].

master log

{"type": "server", "timestamp": "2020-05-21T21:36:30,274Z", "level": "INFO", "component": "o.e.x.s.s.SecurityIndexManager", "cluster.name": "dev", "node.name": "es-master-0", "message": "security index does not exist. Creating [.security-7] with alias [.security]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:36:30,447Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "dev", "node.name": "es-master-0", "message": "[.security-7] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:36:30,458Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "dev", "node.name": "es-master-0", "message": "Cluster health status changed from [YELLOW] to [RED] (reason: [index [.security-7] created]).", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q"  }
{"type": "server", "timestamp": "2020-05-21T21:38:00,568Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "dev", "node.name": "es-master-0", "message": "path: /_security/user/apm_system/_password, params: {pretty=, username=apm_system}", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: [.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update",

I read a bunch of other similar issues on this forum about having data nodes available which I do have.
The only way I made it work was to make the master include data role as well.

3 master nodes

node:
  master: true
  data: true

Why can I not have separate master and data only nodes ?

1 Like
2

Problem solved. Indeed there was not data nodes as the data node configuration was pointing to the master configmap.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.