I spent a lot of time to get a cluster running with one node on 3 different hosts, but all the configuration tips I found didn't fix the problem. I'm using Version 7.9.
This is what I have configured:
cluster.name: elk-dev
node.name: node-3
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost,10.0.2.164
http.port: 9200
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: cert/elk-3.p12
xpack.security.transport.ssl.truststore.path: cert/elk-3.p12
discovery.zen.ping.unicast.hosts: ["10.0.2.199", "10.0.2.242","10.0.2.164"]
Similar configuration for node-1 and node-2
And this is the log until the first errors:
[2021-06-06T13:29:50,161][INFO ][o.e.n.Node ] [node-3] version[7.9.3], pid[20808], build[default/deb/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-1030-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15/15+36-1562]
[2021-06-06T13:29:50,165][INFO ][o.e.n.Node ] [node-3] JVM home [/usr/share/elasticsearch/jdk]
[2021-06-06T13:29:50,165][INFO ][o.e.n.Node ] [node-3] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-14736777617041920716, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-06-06T13:29:53,281][INFO ][o.e.p.PluginsService ] [node-3] loaded module [aggs-matrix-stats]
[2021-06-06T13:29:53,282][INFO ][o.e.p.PluginsService ] [node-3] loaded module [analysis-common]
[2021-06-06T13:29:53,282][INFO ][o.e.p.PluginsService ] [node-3] loaded module [constant-keyword]
[2021-06-06T13:29:53,283][INFO ][o.e.p.PluginsService ] [node-3] loaded module [flattened]
[2021-06-06T13:29:53,283][INFO ][o.e.p.PluginsService ] [node-3] loaded module [frozen-indices]
[2021-06-06T13:29:53,285][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-common]
[2021-06-06T13:29:53,285][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-geoip]
[2021-06-06T13:29:53,286][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-user-agent]
[2021-06-06T13:29:53,286][INFO ][o.e.p.PluginsService ] [node-3] loaded module [kibana]
[2021-06-06T13:29:53,286][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-expression]
[2021-06-06T13:29:53,286][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-mustache]
[2021-06-06T13:29:53,286][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-painless]
[2021-06-06T13:29:53,287][INFO ][o.e.p.PluginsService ] [node-3] loaded module [mapper-extras]
[2021-06-06T13:29:53,287][INFO ][o.e.p.PluginsService ] [node-3] loaded module [parent-join]
[2021-06-06T13:29:53,287][INFO ][o.e.p.PluginsService ] [node-3] loaded module [percolator]
[2021-06-06T13:29:53,287][INFO ][o.e.p.PluginsService ] [node-3] loaded module [rank-eval]
[2021-06-06T13:29:53,288][INFO ][o.e.p.PluginsService ] [node-3] loaded module [reindex]
[2021-06-06T13:29:53,288][INFO ][o.e.p.PluginsService ] [node-3] loaded module [repository-url]
[2021-06-06T13:29:53,288][INFO ][o.e.p.PluginsService ] [node-3] loaded module [search-business-rules]
[2021-06-06T13:29:53,288][INFO ][o.e.p.PluginsService ] [node-3] loaded module [searchable-snapshots]
[2021-06-06T13:29:53,289][INFO ][o.e.p.PluginsService ] [node-3] loaded module [spatial]
[2021-06-06T13:29:53,289][INFO ][o.e.p.PluginsService ] [node-3] loaded module [systemd]
[2021-06-06T13:29:53,289][INFO ][o.e.p.PluginsService ] [node-3] loaded module [tasks]
[2021-06-06T13:29:53,289][INFO ][o.e.p.PluginsService ] [node-3] loaded module [transform]
[2021-06-06T13:29:53,290][INFO ][o.e.p.PluginsService ] [node-3] loaded module [transport-netty4]
[2021-06-06T13:29:53,290][INFO ][o.e.p.PluginsService ] [node-3] loaded module [vectors]
[2021-06-06T13:29:53,290][INFO ][o.e.p.PluginsService ] [node-3] loaded module [wildcard]
[2021-06-06T13:29:53,290][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-analytics]
[2021-06-06T13:29:53,291][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-async]
[2021-06-06T13:29:53,291][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-async-search]
[2021-06-06T13:29:53,291][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-autoscaling]
[2021-06-06T13:29:53,291][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ccr]
[2021-06-06T13:29:53,292][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-core]
[2021-06-06T13:29:53,292][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-data-streams]
[2021-06-06T13:29:53,292][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-deprecation]
[2021-06-06T13:29:53,292][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-enrich]
[2021-06-06T13:29:53,293][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-eql]
[2021-06-06T13:29:53,293][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-graph]
[2021-06-06T13:29:53,293][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-identity-provider]
[2021-06-06T13:29:53,293][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ilm]
[2021-06-06T13:29:53,294][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-logstash]
[2021-06-06T13:29:53,294][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ml]
[2021-06-06T13:29:53,294][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-monitoring]
[2021-06-06T13:29:53,294][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ql]
[2021-06-06T13:29:53,294][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-rollup]
[2021-06-06T13:29:53,295][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-security]
[2021-06-06T13:29:53,295][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-sql]
[2021-06-06T13:29:53,295][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-stack]
[2021-06-06T13:29:53,295][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-voting-only-node]
[2021-06-06T13:29:53,296][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-watcher]
[2021-06-06T13:29:53,296][INFO ][o.e.p.PluginsService ] [node-3] no plugins loaded
[2021-06-06T13:29:53,363][INFO ][o.e.e.NodeEnvironment ] [node-3] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [47.5gb], net total_space [58.1gb], types [ext4]
[2021-06-06T13:29:53,365][INFO ][o.e.e.NodeEnvironment ] [node-3] heap size [2gb], compressed ordinary object pointers [true]
[2021-06-06T13:29:53,585][INFO ][o.e.n.Node ] [node-3] node name [node-3], node ID [xJQYiVPmTYGxCjPccy4Pxg], cluster name [elk-dev]
[2021-06-06T13:29:59,123][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-3] [controller/21004] [Main.cc@114] controller (64 bit): Version 7.9.3 (Build 6c27059cb8397a) Copyright (c) 2020 Elasticsearch BV
[2021-06-06T13:29:59,932][INFO ][o.e.x.s.a.s.FileRolesStore] [node-3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-06-06T13:30:01,079][INFO ][o.e.t.NettyAllocator ] [node-3] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=256kb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=1mb}]
[2021-06-06T13:30:01,162][INFO ][o.e.d.DiscoveryModule ] [node-3] using discovery type [zen] and seed hosts providers [settings]
[2021-06-06T13:30:01,746][WARN ][o.e.g.DanglingIndicesState] [node-3] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-06-06T13:30:02,234][INFO ][o.e.n.Node ] [node-3] initialized
[2021-06-06T13:30:02,235][INFO ][o.e.n.Node ] [node-3] starting ...
[2021-06-06T13:30:02,359][INFO ][o.e.t.TransportService ] [node-3] publish_address {10.0.2.164:9300}, bound_addresses {127.0.0.1:9300}, {10.0.2.164:9300}
[2021-06-06T13:30:02,966][INFO ][o.e.b.BootstrapChecks ] [node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-06-06T13:30:02,969][INFO ][o.e.c.c.Coordinator ] [node-3] cluster UUID [b4A-TuEbQHigHpdiCYnukA]
[2021-06-06T13:30:03,186][INFO ][o.e.c.s.MasterService ] [node-3] elected-as-master ([1] nodes joined)[{node-3}{xJQYiVPmTYGxCjPccy4Pxg}{HjKO6pWGQNmePKAuQnZZBA}{10.0.2.164}{10.0.2.164:9300}{dilmrt}{ml.machine_memory=4064817152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 41, version: 2195, delta: master node changed {previous [], current [{node-3}{xJQYiVPmTYGxCjPccy4Pxg}{HjKO6pWGQNmePKAuQnZZBA}{10.0.2.164}{10.0.2.164:9300}{dilmrt}{ml.machine_memory=4064817152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2021-06-06T13:30:03,550][INFO ][o.e.c.s.ClusterApplierService] [node-3] master node changed {previous [], current [{node-3}{xJQYiVPmTYGxCjPccy4Pxg}{HjKO6pWGQNmePKAuQnZZBA}{10.0.2.164}{10.0.2.164:9300}{dilmrt}{ml.machine_memory=4064817152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 41, version: 2195, reason: Publication{term=41, version=2195}
[2021-06-06T13:30:04,911][INFO ][o.e.h.AbstractHttpServerTransport] [node-3] publish_address {10.0.2.164:9200}, bound_addresses {127.0.0.1:9200}, {10.0.2.164:9200}
[2021-06-06T13:30:04,911][INFO ][o.e.n.Node ] [node-3] started
[2021-06-06T13:30:05,424][ERROR][o.e.x.s.a.e.ReservedRealm] [node-3] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at
org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:181) ~[x-pack-security-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:525) [x-pack-security-7.9.3.jar:7.9.3]
But running
GET _cluster/health
I always get:
{
"cluster_name" : "elk-dev",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 37,
"active_shards" : 37,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 7,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 84.0909090909091
}
What am I doing wrong? Any help would be very appreciated
Thanks!
