Unable to setup first time password for the new cluster

Elastic Stack Elasticsearch
1

I install brand new elasticsearch 7.1.1 cluster using saltstack.
I start from 3 master nodes and they properly provisioned and from cluster.
After that i want to set-up passwords for built-in users, before provisioning data nodes by running

Blockquote
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -u 'http://127.0.0.1:9200'
/Blockquote

and this is what i receive:

Blockquote
root@es-master1:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -u 'http://127.0.0.1:9200'
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y
Connection failure to: http://127.0.0.1:9200/_security/user/apm_system/_password?pretty failed: Read timed out
ERROR: Failed to set password for user [apm_system].
Blockquote

Master node configuration:
#======================== Elasticsearch Configuration =========================
cluster.name: ad-test
node.name: es-master1
node.master: True
node.data: False
node.ingest: True
path.logs: /var/log/elasticsearch
path.data: /var/lib/elasticsearch
xpack.security.enabled: True
xpack.monitoring.collection.enabled: True
xpack.security.transport.ssl.enabled: True
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/ad-test-certificate.p12
xpack.security.transport.ssl.truststore.path: certs/ad-test-certificate.p12
bootstrap.memory_lock: True
network.host: 0.0.0.0
cluster.initial_master_nodes: [172.31.192.78, 172.31.192.91, 172.31.192.103]
discovery.seed_hosts: [172.31.192.78, 172.31.192.91, 172.31.192.103]

Before i run command to setup default users password i see the following errors in cluster.log

Blockquote
[2019-07-04T15:21:26,745][WARN ][o.e.x.m.e.l.LocalExporter] [es-master1] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-7-2019.07.04][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-7-2019.07.04][0]] containing [4] requests]]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:928) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-7-2019.07.04][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-7-2019.07.04][0]] containing [4] requests]
Blockquote

2

Hi @dumkaz,

Could you please post the output for the GET _cluster/health API

Thanks and Regards,
Yogesh Gaikwad

3

@Yogesh_Gaikwad
t won`t work, as something went wrong with setting password for default users.
but anyway:
curl -XGET http://127.0.0.1:9200/_cluster/health
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

4

using interactive mode on brand new cluster gives the same error
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:

Connection failure to: http://127.0.0.1:9200/_security/user/apm_system/_password?pretty failed: Read timed out

ERROR: Failed to set password for user [apm_system].

5

I think I have found the problem: redeploying right now with data nodes

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.