Error when try to create a new cluster elasticsearch


(piter) #1

I would like to create an elastic cluster with 3 nodes to training purpose.
This is my conf for node1

[elastic@localhost config]$ more elasticsearch.yml
    # ======================== Elasticsearch Configuration =========================
    #
    # NOTE: Elasticsearch comes with reasonable defaults for most settings.
    #       Before you set out to tweak and tune the configuration, make sure you
    #       understand what are you trying to accomplish and the consequences.
    #
    # The primary way of configuring a node is via this file. This template lists
    # the most important settings you may want to configure for a production cluster.
    #
    # Please consult the documentation for further information on configuration options:
    # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
    #
    # ---------------------------------- Cluster -----------------------------------
    #
    # Use a descriptive name for your cluster:
    #
    cluster.name: elastic
    #
    # ------------------------------------ Node ------------------------------------
    #
    # Use a descriptive name for the node:
    #
    node.name: ${HOSTNAME}
    #
    # Add custom attributes to the node:
    #
    node.attr.rack: r1
    #
    # ----------------------------------- Paths ------------------------------------
    #
    # Path to directory where to store the data (separate multiple locations by comma):
    #
    #path.data: /path/to/data
    #
    # Path to log files:
    #
    path.logs: /var/log/elastic/
    #
    # ----------------------------------- Memory -----------------------------------
    #
    # Lock the memory on startup:
    #
    #bootstrap.memory_lock: true
    #
    # Make sure that the heap size is set to about half the memory available
    # on the system and that the owner of the process is allowed to use this
    # limit.
    #
    # Elasticsearch performs poorly when the system is swapping the memory.
    #
    # ---------------------------------- Network -----------------------------------
    #
    # Set the bind address to a specific IP (IPv4 or IPv6):
    #
    network.host: 192.168.28.101
    #
    # Set a custom port for HTTP:
    #
    http.port: 9200
    transport.tcp.port: 9500
    #
    # For more information, consult the network module documentation.
    #
    # --------------------------------- Discovery ----------------------------------
    #
    # Pass an initial list of hosts to perform discovery when new node is started:
    # The default list of hosts is ["127.0.0.1", "[::1]"]
    #
    discovery.zen.ping.unicast.hosts: ["192.168.28.102:9500","192.168.28.103:9500"]
    #
    # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
    #
    discovery.zen.minimum_master_nodes: 2
    #
    # For more information, consult the zen discovery module documentation.
    #
    # ---------------------------------- Gateway -----------------------------------
    #
    # Block initial recovery after a full cluster restart until N nodes are started:
    #
    #gateway.recover_after_nodes: 3
    #
    # For more information, consult the gateway module documentation.
    #
    # ---------------------------------- Various -----------------------------------
    #
    # Require explicit names when deleting indices:
    #
    #action.destructive_requires_name: true

The cluster don't go up:

[2018-10-31T19:33:27,971][INFO ][o.e.d.z.ZenDiscovery ] [localhost.localdomain] failed to send join request to master [{node3}{1O65OSvATQKsniFvSZXCSQ}{ARW4IYeASBypo_1VyJyAEg}{192.168.28.103}{192.168.28.103:9500}{ml.machine_memory=2922430464, rack=r3, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node3][192.168.28.103:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node3}{1O65OSvATQKsniFvSZXCSQ}{ARW4IYeASBypo_1VyJyAEg}{192.168.28.103}{192.168.28.103:9500}{ml.machine_memory=2922430464, rack=r3, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2018-10-31T19:33:31,286][INFO ][o.e.d.z.ZenDiscovery ]


(Kanagat) #2

HI @elk2
can you check that network connection open between this servers?
and firewall not block connections.
can you send result of commands:

from server1:
curl server2:9200
curl server3:9200

and according from other servers


(piter) #3

I disabled iptables on all nodes.
Telnet on port 9200 and 9500 work on all nodes after elasticsearch starts


(Kanagat) #4

can you send logs from server1, server2, server3?


(piter) #5

The servers are identically because I cloned the node1 server , changed ip adress in elastic conf and interface network.

Node3:

[2018-10-31T19:45:41,099][INFO ][o.e.d.z.ZenDiscovery ] [node3] failed to send join request to master [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node2][192.168.28.102:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2018-10-31T19:45:44,365][INFO ][o.e.d.z.ZenDiscovery ] [node3] failed to send join request to master [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node2][192.168.28.102:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
^C

Node2

[2018-10-31T19:49:31,058][INFO ][o.e.d.z.ZenDiscovery ] [node2] failed to send join request to master [{localhost.localdomain}{1O65OSvATQKsniFvSZXCSQ}{MNOEYeHhTxu7jvi_q_Ca-g}{192.168.28.101}{192.168.28.101:9500}{ml.machine_memory=2922434560, rack=r1, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[localhost.localdomain][192.168.28.101:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{localhost.localdomain}{1O65OSvATQKsniFvSZXCSQ}{MNOEYeHhTxu7jvi_q_Ca-g}{192.168.28.101}{192.168.28.101:9500}{ml.machine_memory=2922434560, rack=r1, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2018-10-31T19:49:37,256][INFO ][o.e.d.z.ZenDiscovery ] [node2] failed to send join request to master [{localhost.localdomain}{1O65OSvATQKsniFvSZXCSQ}{MNOEYeHhTxu7jvi_q_Ca-g}{192.168.28.101}{192.168.28.101:9500}{ml.machine_memory=2922434560, rack=r1, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[localhost.localdomain][192.168.28.101:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{localhost.localdomain}{1O65OSvATQKsniFvSZXCSQ}{MNOEYeHhTxu7jvi_q_Ca-g}{192.168.28.101}{192.168.28.101:9500}{ml.machine_memory=2922434560, rack=r1, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times

Node1

[2018-10-31T19:57:21,369][INFO ][o.e.d.z.ZenDiscovery ] [localhost.localdomain] failed to send join request to master [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node2][192.168.28.102:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2018-10-31T19:57:27,573][INFO ][o.e.d.z.ZenDiscovery ] [localhost.localdomain] failed to send join request to master [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node2][192.168.28.102:9500][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node2}{1O65OSvATQKsniFvSZXCSQ}{6tdM3OwoSD-N61JPqFlACA}{192.168.28.102}{192.168.28.102:9500}{ml.machine_memory=2922434560, rack=r2, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times


(piter) #6

Any suggest??

Update:

I modified elasticsearch.yml with network.host:0.0.0.0 and removed custom port.

This is log:

[elastic@localhost config]$ ../bin/elasticsearch
[2018-10-31T23:44:31,571][INFO ][o.e.n.Node ] [node3] initializing ...
[2018-10-31T23:44:31,796][INFO ][o.e.e.NodeEnvironment ] [node3] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [34.3gb], net total_space [35.9gb], types [rootfs]
[2018-10-31T23:44:31,797][INFO ][o.e.e.NodeEnvironment ] [node3] heap size [1007.3mb], compressed ordinary object pointers [true]
[2018-10-31T23:44:31,799][INFO ][o.e.n.Node ] [node3] node name [node3], node ID [1O65OSvATQKsniFvSZXCSQ]
[2018-10-31T23:44:31,800][INFO ][o.e.n.Node ] [node3] version[6.4.2], pid[2807], build[default/tar/04711c2/2018-09-26T13:34:09.098244Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2018-10-31T23:44:31,801][INFO ][o.e.n.Node ] [node3] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.5QFuCPZJ, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/elastic/elasticsearch-6.4.2, -Des.path.conf=/home/elastic/elasticsearch-6.4.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-10-31T23:44:36,572][INFO ][o.e.p.PluginsService ] [node3] loaded module [aggs-matrix-stats]
[2018-10-31T23:44:36,573][INFO ][o.e.p.PluginsService ] [node3] loaded module [analysis-common]
[2018-10-31T23:44:36,573][INFO ][o.e.p.PluginsService ] [node3] loaded module [ingest-common]
[2018-10-31T23:44:36,573][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-expression]
[2018-10-31T23:44:36,576][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-mustache]
[2018-10-31T23:44:36,576][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-painless]
[2018-10-31T23:44:36,576][INFO ][o.e.p.PluginsService ] [node3] loaded module [mapper-extras]
[2018-10-31T23:44:36,576][INFO ][o.e.p.PluginsService ] [node3] loaded module [parent-join]
[2018-10-31T23:44:36,576][INFO ][o.e.p.PluginsService ] [node3] loaded module [percolator]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [rank-eval]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [reindex]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [repository-url]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [transport-netty4]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [tribe]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-core]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-deprecation]
[2018-10-31T23:44:36,577][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-graph]
[2018-10-31T23:44:36,578][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-logstash]
[2018-10-31T23:44:36,578][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-ml]
[2018-10-31T23:44:36,578][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-monitoring]
[2018-10-31T23:44:36,579][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-rollup]
[2018-10-31T23:44:36,579][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-security]
[2018-10-31T23:44:36,579][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-sql]
[2018-10-31T23:44:36,579][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-upgrade]
[2018-10-31T23:44:36,585][INFO ][o.e.p.PluginsService ] [node3] loaded module [x-pack-watcher]
[2018-10-31T23:44:36,586][INFO ][o.e.p.PluginsService ] [node3] no plugins loaded
[2018-10-31T23:44:46,259][INFO ][o.e.x.s.a.s.FileRolesStore] [node3] parsed [0] roles from file [/home/elastic/elasticsearch-6.4.2/config/roles.yml]
[2018-10-31T23:44:48,414][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/2862] [Main.cc@109] controller (64 bit): Version 6.4.2 (Build 660eefe6f2ea55) Copyright (c) 2018 Elasticsearch BV
[2018-10-31T23:44:49,019][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-10-31T23:44:49,624][INFO ][o.e.d.DiscoveryModule ] [node3] using discovery type [zen]
[2018-10-31T23:44:51,371][INFO ][o.e.n.Node ] [node3] initialized
[2018-10-31T23:44:51,372][INFO ][o.e.n.Node ] [node3] starting ...
[2018-10-31T23:44:51,768][INFO ][o.e.t.TransportService ] [node3] publish_address {192.168.28.103:9300}, bound_addresses {[::]:9300}
[2018-10-31T23:44:51,827][INFO ][o.e.b.BootstrapChecks ] [node3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-10-31T23:44:54,971][WARN ][o.e.d.z.ZenDiscovery ] [node3] not enough master nodes discovered during pinging (found [[Candidate{node={node3}{1O65OSvATQKsniFvSZXCSQ}{6uACxm0YTtasWXRwtrKvUQ}{192.168.28.103}{192.168.28.103:9300}{ml.machine_memory=2922430464, rack=r3, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-31T23:44:57,977][WARN ][o.e.d.z.ZenDiscovery ] [node3] not enough master nodes discovered during pinging (found [[Candidate{node={node3}{1O65OSvATQKsniFvSZXCSQ}{6uACxm0YTtasWXRwtrKvUQ}{192.168.28.103}{192.168.28.103:9300}{ml.machine_memory=2922430464, rack=r3, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-31T23:45:00,982][WARN ][o.e.d.z.ZenDiscovery ] [node3] not enough master nodes discovered during pinging (found [[Candidate{node={node3}{1O65OSvATQKsniFvSZXCSQ}{6uACxm0YTtasWXRwtrKvUQ}{192.168.28.103}{192.168.28.103:9300}{ml.machine_memory=2922430464, rack=r3, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.