Im running elasticsearch 7.6.2 Basic license
3 master nodes
cluster:
name: dev
node:
master: true
data: false
ingest: false
network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true
3 data nodes
cluster.name: dev
node:
master: false
data: true
ingest: true
network.host: "0.0.0.0"
cluster.initial_master_nodes: "es-master-0,es-master-1,es-master-2"
discovery.seed_hosts: "es-master-0.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-1.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300,es-master-2.elasticsearch-master.elasticsearch-logging.svc.cluster.local:9300"
xpack.security.enabled: true
If I set, the cluster comes up fine
xpack.security.enabled: false
When I set, the clustering seems fine.
xpack.security.enabled: true
But when I run
bin/elasticsearch-setup-passwords auto -b --verbose
Cluster is green, but
Trying user password change call http://10.1.46.122:9200/_security/user/apm_system/_password?pretty
Connection failure to: http://10.1.46.122:9200/_security/user/apm_system/_password?pretty failed: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:284)
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:310)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:937)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:245)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:285)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:344)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:119)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changeUserPassword(SetupPasswordTool.java:462)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.changePasswords(SetupPasswordTool.java:523)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:143)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:107)
ERROR: Failed to set password for user [apm_system].
master log
{"type": "server", "timestamp": "2020-05-21T21:36:30,274Z", "level": "INFO", "component": "o.e.x.s.s.SecurityIndexManager", "cluster.name": "dev", "node.name": "es-master-0", "message": "security index does not exist. Creating [.security-7] with alias [.security]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" }
{"type": "server", "timestamp": "2020-05-21T21:36:30,447Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "dev", "node.name": "es-master-0", "message": "[.security-7] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" }
{"type": "server", "timestamp": "2020-05-21T21:36:30,458Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "dev", "node.name": "es-master-0", "message": "Cluster health status changed from [YELLOW] to [RED] (reason: [index [.security-7] created]).", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" }
{"type": "server", "timestamp": "2020-05-21T21:38:00,568Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "dev", "node.name": "es-master-0", "message": "path: /_security/user/apm_system/_password, params: {pretty=, username=apm_system}", "cluster.uuid": "knZ2lyedTi-fhFhtpena0w", "node.id": "ZTs2hhJeTnS74li2OWQd_Q" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: [.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update",
I read a bunch of other similar issues on this forum about having data nodes available which I do have.
The only way I made it work was to make the master include data role as well.
3 master nodes
node:
master: true
data: true
Why can I not have separate master and data only nodes ?