I install brand new elasticsearch 7.1.1 cluster using saltstack.
I start from 3 master nodes and they properly provisioned and from cluster.
After that i want to set-up passwords for built-in users, before provisioning data nodes by running
Blockquote
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -u 'http://127.0.0.1:9200'
/Blockquote
and this is what i receive:
Blockquote
root@es-master1:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -u 'http://127.0.0.1:9200'
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y
Connection failure to: http://127.0.0.1:9200/_security/user/apm_system/_password?pretty failed: Read timed out
ERROR: Failed to set password for user [apm_system].
Blockquote
Master node configuration:
#======================== Elasticsearch Configuration =========================
cluster.name: ad-test
node.name: es-master1
node.master: True
node.data: False
node.ingest: True
path.logs: /var/log/elasticsearch
path.data: /var/lib/elasticsearch
xpack.security.enabled: True
xpack.monitoring.collection.enabled: True
xpack.security.transport.ssl.enabled: True
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/ad-test-certificate.p12
xpack.security.transport.ssl.truststore.path: certs/ad-test-certificate.p12
bootstrap.memory_lock: True
network.host: 0.0.0.0
cluster.initial_master_nodes: [172.31.192.78, 172.31.192.91, 172.31.192.103]
discovery.seed_hosts: [172.31.192.78, 172.31.192.91, 172.31.192.103]
Before i run command to setup default users password i see the following errors in cluster.log
Blockquote
[2019-07-04T15:21:26,745][WARN ][o.e.x.m.e.l.LocalExporter] [es-master1] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-7-2019.07.04][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-7-2019.07.04][0]] containing [4] requests]]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:928) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-7-2019.07.04][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-7-2019.07.04][0]] containing [4] requests]
Blockquote