I want to setup cluster mode between 2 linux servers
One of them will be master and data node and other one will only be a data node.
Whenever i try to setup any external IP/interfaces to transport.host, elasticsearch bootstrap checks are triggered and I am no longer able to start the elastic service.
And if transport.host is made to listen on localhost by default setting, other data-node unable to fetch the master node
Hello @HiteshSingh , please share the elasticsearch.yml file from master/data nodes and error logs from data node
Master-node config:
# ======================== Elasticsearch Configuration =========================
#
# ---------------------------------- Cluster -----------------------------------
#
#cluster.name: my-application
cluster.name: my-app
#
# ------------------------------------ Node ------------------------------------
node.name: myapp-dev-node-1
node.roles: [master, data]
discovery.seed_hosts: ["<linux_master_server_ip>:9300","<linux_data_server_ip>:9300"]
#elastic elasticsearch configuration
path.data: /data/elastic/search-data
path.logs: /data/elastic/search-logs
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["myapp-dev-node-1"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0
#network.host: _site_
network.host: <linux_server_ip>
http.port: 9090
transport.host: localhost
transport.port: 9300
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
Data-node config:
# ======================== Elasticsearch Configuration =========================
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-app
node.name: myapp-dev-node-2
node.roles: [data]
discovery.seed_hosts: ["<linux_master_server_ip>:9300","<linux_data_server_ip>:9300"]
#elastic elasticsearch configuration
path.data: /data/elastic/search-data
path.logs: /data/elastic/search-logs
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["myapp-dev-node-1"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0
network.host: <linux_data_server_ip>
http.port: 9090
transport.host: localhost
transport.port: 9300
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
Data node error logs:
[2023-08-21T07:47:21,690][INFO ][o.e.p.PluginsService ] [palms-dev-node-2] loaded module [x-pack-watcher]
[2023-08-21T07:47:21,690][INFO ][o.e.p.PluginsService ] [palms-dev-node-2] loaded module [x-pack-write-load-forecaster]
[2023-08-21T07:47:21,690][INFO ][o.e.p.PluginsService ] [palms-dev-node-2] no plugins loaded
[2023-08-21T07:47:23,914][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-2] using [1] data paths, mounts [[/data (/dev/mapper/datavg-lv_data)]], net usable_space [135.7gb], net total_space [145.5gb], types [ext4]
[2023-08-21T07:47:23,915][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-2] heap size [7.6gb], compressed ordinary object pointers [true]
[2023-08-21T07:47:23,956][INFO ][o.e.n.Node ] [palms-dev-node-2] node name [palms-dev-node-2], node ID [JJ19xi74SAmF220q2PccLQ], cluster name [palms-dev-els], roles [data]
[2023-08-21T07:47:26,560][INFO ][o.e.x.s.Security ] [palms-dev-node-2] Security is enabled
[2023-08-21T07:47:26,866][INFO ][o.e.x.s.a.s.FileRolesStore] [palms-dev-node-2] parsed [0] roles from file [/data/PALMS_ELS/elastic/elasticsearch/config/roles.yml]
[2023-08-21T07:47:27,245][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [palms-dev-node-2] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2023-08-21T07:47:27,328][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [palms-dev-node-2] [controller/362110] [Main.cc@123] controller (64 bit): Version 8.6.2 (Build 0d41528b670ce1) Copyright (c) 2023 Elasticsearch BV
[2023-08-21T07:47:27,904][INFO ][o.e.t.n.NettyAllocator ] [palms-dev-node-2] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-08-21T07:47:27,927][INFO ][o.e.i.r.RecoverySettings ] [palms-dev-node-2] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-08-21T07:47:27,945][DEBUG][o.e.d.SettingsBasedSeedHostsProvider] [palms-dev-node-2] using initial hosts [narvlii15373.xmp.net.intra]
[2023-08-21T07:47:27,962][INFO ][o.e.d.DiscoveryModule ] [palms-dev-node-2] using discovery type [multi-node] and seed hosts providers [settings]
[2023-08-21T07:47:29,128][INFO ][o.e.n.Node ] [palms-dev-node-2] initialized
[2023-08-21T07:47:29,129][INFO ][o.e.n.Node ] [palms-dev-node-2] starting ...
[2023-08-21T07:47:29,148][INFO ][o.e.x.s.c.f.PersistentCache] [palms-dev-node-2] persistent cache index loaded
[2023-08-21T07:47:29,149][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [palms-dev-node-2] deprecation component started
[2023-08-21T07:47:29,236][INFO ][o.e.t.TransportService ] [palms-dev-node-2] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-08-21T07:47:29,358][WARN ][o.e.b.BootstrapChecks ] [palms-dev-node-2] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2023-08-21T07:47:29,360][DEBUG][o.e.d.SeedHostsResolver ] [palms-dev-node-2] using max_concurrent_resolvers [10], resolver timeout [5s]
[2023-08-21T07:47:29,360][INFO ][o.e.c.c.ClusterBootstrapService] [palms-dev-node-2] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [palms-dev-node-1]
[2023-08-21T07:47:39,372][WARN ][o.e.c.c.ClusterFormationFailureHelper] [palms-dev-node-2] master not discovered yet: have discovered [{palms-dev-node-2}{JJ19xi74SAmF220q2PccLQ}{iKDLL49QT3y6BzuZgb9m4A}{palms-dev-node-2}{localhost}{127.0.0.1:9300}{d}]; discovery will continue using [10.107.79.73:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
This needs to be 0.0.0.0 or the private ip address of the server, if you are using localhost it will not be accessible by external machines, so your nodes won't be able to talk with each other.
Try to use this in your data node config and run again -
Instead of this
cluster.initial_master_nodes: ["myapp-dev-node-1"]
Try this
cluster.initial_master_nodes: ["myapp-dev-node-2"]
Also check whether connectivity is there between these VM's on mentioned transport port ?
I have already tried this but as you can see in my question whenever i try to keep transport.host to any value other than localhost it gives me this error:
[2023-08-21T08:49:46,303][INFO ][o.e.p.PluginsService ] [palms-dev-node-1] loaded module [x-pack-write-load-forecaster]
[2023-08-21T08:49:46,303][INFO ][o.e.p.PluginsService ] [palms-dev-node-1] no plugins loaded
[2023-08-21T08:49:48,790][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-1] using [1] data paths, mounts [[/data (/dev/mapper/datavg-lv_data)]], net usable_space [952.3gb], net total_space [999gb], types [ext4]
[2023-08-21T08:49:48,791][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-1] heap size [31gb], compressed ordinary object pointers [true]
[2023-08-21T08:49:48,800][INFO ][o.e.n.Node ] [palms-dev-node-1] node name [palms-dev-node-1], node ID [aGLeOpbUQdCkSZaul4AWww], cluster name [palms-dev-els], roles [master, data]
[2023-08-21T08:49:52,161][INFO ][o.e.x.s.Security ] [palms-dev-node-1] Security is enabled
[2023-08-21T08:49:52,462][INFO ][o.e.x.s.a.s.FileRolesStore] [palms-dev-node-1] parsed [0] roles from file [/data/PALMS_ELS/elastic/elasticsearch/config/roles.yml]
[2023-08-21T08:49:52,824][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [palms-dev-node-1] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2023-08-21T08:49:52,916][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [palms-dev-node-1] [controller/1287485] [Main.cc@123] controller (64 bit): Version 8.6.2 (Build 0d41528b670ce1) Copyright (c) 2023 Elasticsearch BV
[2023-08-21T08:49:53,516][INFO ][o.e.t.n.NettyAllocator ] [palms-dev-node-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2023-08-21T08:49:53,544][INFO ][o.e.i.r.RecoverySettings ] [palms-dev-node-1] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-08-21T08:49:53,565][DEBUG][o.e.d.SettingsBasedSeedHostsProvider] [palms-dev-node-1] using initial hosts [narvlii15373.xmp.net.intra:9300, narvlii15490.xmp.net.intra:9300, 127.0.0.1]
[2023-08-21T08:49:53,583][INFO ][o.e.d.DiscoveryModule ] [palms-dev-node-1] using discovery type [multi-node] and seed hosts providers [settings]
[2023-08-21T08:49:54,851][INFO ][o.e.n.Node ] [palms-dev-node-1] initialized
[2023-08-21T08:49:54,852][INFO ][o.e.n.Node ] [palms-dev-node-1] starting ...
[2023-08-21T08:49:54,867][INFO ][o.e.x.s.c.f.PersistentCache] [palms-dev-node-1] persistent cache index loaded
[2023-08-21T08:49:54,869][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [palms-dev-node-1] deprecation component started
[2023-08-21T08:49:54,974][INFO ][o.e.t.TransportService ] [palms-dev-node-1] publish_address {10.107.79.73:9300}, bound_addresses {[::]:9300}
[2023-08-21T08:49:55,083][INFO ][o.e.b.BootstrapChecks ] [palms-dev-node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-08-21T08:49:55,094][INFO ][o.e.n.Node ] [palms-dev-node-1] stopping ...
[2023-08-21T08:49:55,158][INFO ][o.e.n.Node ] [palms-dev-node-1] stopped
[2023-08-21T08:49:55,158][INFO ][o.e.n.Node ] [palms-dev-node-1] closing ...
[2023-08-21T08:49:55,168][INFO ][o.e.n.Node ] [palms-dev-node-1] closed
[2023-08-21T08:49:55,170][INFO ][o.e.x.m.p.NativeController] [palms-dev-node-1] Native controller process has stopped - no new native processes can be started
tried this as well but still same data node error pasted in above trail.
Also, I have the connectivity between the hosts on port 9300.
Tried this from data node:
nc -v 10.107.79.73 9300
Ncat: Version 7.91 ( https://nmap.org/ncat )
Ncat: Connected to 10.107.79.73:9300.
This will not work, on a 2 node cluster only one node can be the master, in this case it is the myapp-dev-node-1, this is the node that needs to be used in cluster.initial_master_nodes.
From which node is this log? There is no error/warn lines on this log, just INFO.
You need to fix this, the node needs to pass all the bootstrap checks. Check the documentation.
Alright @leandrojmp , I will try to increase this VM count and inform.
Meanwhile i do this, Can you tell me what all IP's discovery.seed_hosts should have in my master and data node.
It is explained in the documentation, discovery.seed_hosts needs to have only the master-elegible nodes, so in your case just the first node.
I tried increasing VM count and that error no longer pops up in the logs but still data node unable to join the cluster:
Data node logs:
[2023-08-21T09:42:28,428][INFO ][o.e.p.PluginsService ] [palms-dev-node-2] loaded module [x-pack-write-load-forecaster]
[2023-08-21T09:42:28,428][INFO ][o.e.p.PluginsService ] [palms-dev-node-2] no plugins loaded
[2023-08-21T09:42:30,819][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-2] using [1] data paths, mounts [[/data (/dev/mapper/datavg-lv_data)]], net usable_space [135.7gb], net total_space [145.5gb], types [ext4]
[2023-08-21T09:42:30,820][INFO ][o.e.e.NodeEnvironment ] [palms-dev-node-2] heap size [7.6gb], compressed ordinary object pointers [true]
[2023-08-21T09:42:30,831][INFO ][o.e.n.Node ] [palms-dev-node-2] node name [palms-dev-node-2], node ID [MXNbQYj5RYirU4ZbsMwBRA], cluster name [palms-dev-els], roles [data]
[2023-08-21T09:42:33,927][INFO ][o.e.x.s.Security ] [palms-dev-node-2] Security is enabled
[2023-08-21T09:42:34,236][INFO ][o.e.x.s.a.s.FileRolesStore] [palms-dev-node-2] parsed [0] roles from file [/data/PALMS_ELS/elastic/elasticsearch/config/roles.yml]
[2023-08-21T09:42:34,572][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [palms-dev-node-2] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2023-08-21T09:42:34,654][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [palms-dev-node-2] [controller/394338] [Main.cc@123] controller (64 bit): Version 8.6.2 (Build 0d41528b670ce1) Copyright (c) 2023 Elasticsearch BV
[2023-08-21T09:42:35,241][INFO ][o.e.t.n.NettyAllocator ] [palms-dev-node-2] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-08-21T09:42:35,266][INFO ][o.e.i.r.RecoverySettings ] [palms-dev-node-2] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-08-21T09:42:35,286][DEBUG][o.e.d.SettingsBasedSeedHostsProvider] [palms-dev-node-2] using initial hosts [narvlii15373.xmp.net.intra:9300]
[2023-08-21T09:42:35,304][INFO ][o.e.d.DiscoveryModule ] [palms-dev-node-2] using discovery type [multi-node] and seed hosts providers [settings]
[2023-08-21T09:42:36,529][INFO ][o.e.n.Node ] [palms-dev-node-2] initialized
[2023-08-21T09:42:36,530][INFO ][o.e.n.Node ] [palms-dev-node-2] starting ...
[2023-08-21T09:42:36,541][INFO ][o.e.x.s.c.f.PersistentCache] [palms-dev-node-2] persistent cache index loaded
[2023-08-21T09:42:36,542][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [palms-dev-node-2] deprecation component started
[2023-08-21T09:42:36,638][INFO ][o.e.t.TransportService ] [palms-dev-node-2] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-08-21T09:42:36,745][DEBUG][o.e.d.SeedHostsResolver ] [palms-dev-node-2] using max_concurrent_resolvers [10], resolver timeout [5s]
[2023-08-21T09:42:36,746][INFO ][o.e.c.c.ClusterBootstrapService] [palms-dev-node-2] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [palms-dev-node-1]
[2023-08-21T09:42:46,760][WARN ][o.e.c.c.ClusterFormationFailureHelper] [palms-dev-node-2] master not discovered yet: have discovered [{palms-dev-node-2}{MXNbQYj5RYirU4ZbsMwBRA}{TU3AZ9DMS_eXTbzWbKGunQ}{palms-dev-node-2}{localhost}{127.0.0.1:9300}{d}]; discovery will continue using [10.107.79.73:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-08-21T09:42:56,762][WARN ][o.e.c.c.ClusterFormationFailureHelper] [palms-dev-node-2] master not discovered yet: have discovered [{palms-dev-node-2}{MXNbQYj5RYirU4ZbsMwBRA}{TU3AZ9DMS_eXTbzWbKGunQ}{palms-dev-node-2}{localhost}{127.0.0.1:9300}{d}]; discovery will continue using [10.107.79.73:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-08-21T09:43:06,761][WARN ][o.e.n.Node ] [palms-dev-node-2] timed out while waiting for initial discovery state - timeout: 30s
[2023-08-21T09:43:06,763][WARN ][o.e.c.c.ClusterFormationFailureHelper] [palms-dev-node-2] master not discovered yet: have discovered [{palms-dev-node-2}{MXNbQYj5RYirU4ZbsMwBRA}{TU3AZ9DMS_eXTbzWbKGunQ}{palms-dev-node-2}{localhost}{127.0.0.1:9300}{d}]; discovery will continue using [10.107.79.73:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-08-21T09:43:06,766][INFO ][o.e.h.AbstractHttpServerTransport] [palms-dev-node-2] publish_address {10.107.78.128:9300}, bound_addresses {10.107.78.128:9300}
[2023-08-21T09:43:06,767][INFO ][o.e.n.Node ] [palms-dev-node-2] started {palms-dev-node-2}{MXNbQYj5RYirU4ZbsMwBRA}{TU3AZ9DMS_eXTbzWbKGunQ}{palms-dev-node-2}{localhost}{127.0.0.1:9300}{d}{xpack.installed=true}
[2023-08-21T09:43:06,883][DEBUG][o.e.d.PeerFinder ] [palms-dev-node-2] address [10.107.79.73:9300], node [null], requesting [false] discovery result
org.elasticsearch.transport.ConnectTransportException: [][10.107.79.73:9300] handshake_timeout[30s]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:71) ~[elasticsearch-8.6.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:850) ~[elasticsearch-8.6.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Is transport.host set to 0.0.0.0 or the server private ip address in both nodes?
What do you have in the first node logs?
We need to increase vm.max_map_count to 262144 and this resolved my issue
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.