Es 5 node cannot join the master node


(mabrouk) #1

I configured two-node cluster with one master node. It's Red Hat servers. Unfortunately second node cannot join to the master node. Here is configuration of master node:

Use a descriptive name for the node:

node.name: "nodetwo"
node.master: true
node.data: false
#network.host: 192.168.10.100
#transport.host: 192.168.10.1

Add custom attributes to the node:

#node.attr.rack: r1
discovery.zen.ping.unicast.hosts: ["192.168.10.100", "192.168.10.150"]

----------------------------------- Paths ------------------------------------


(Jason Tedor) #2

It looks like you're setting the transport address to a different address than the you set in discovery.zen.ping.unicast.hosts (I say look like because you did not show both sides). Elasticsearch uses the transport address for internode communication, so you need to ensure the transport addresses and the addresses that you use in discovery.zen.ping.unicast.hosts are the same.


(mabrouk) #3

the second elasticsearch.yml :# ---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: prod

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: "nodeone"
node.master: false
node.data: true
discovery.zen.ping.unicast.hosts: ["192.168.10.100", "192.168.10.150"]

Add custom attributes to the node:

#node.attr.rack: r1


(Jason Tedor) #4

Right, so as I mentioned, the problem is that you have specified a different transport address than you specified in discovery.zen.ping.unicast.hosts.


(mabrouk) #5

Then what should I do?
Is what I add transport.host: 192.168.10.150


(Jason Tedor) #6

As I've explained, you set the transport address to 192.168.10.1 on your master. This is the address that you must use for internode communication. This means that your discovery.zen.ping.unicast.hosts should include 192.168.10.1 instead of 192.168.10.100 as there is no node listening for transport requests on 192.168.10.100.


(mabrouk) #7

I have two machines the master has the address 192.168.10.150 and the slave 192.168.10.100
I do not use the address 192.168.10.1
elasticsearch.yml

cluster.name: prod
node.name: "nodeone"
node.master: false
node.data: true
discovery.zen.ping.unicast.hosts: ["192.168.10.100", "192.168.10.150"]
transport.host: 192.168.10.150 //master Address
#//transport.host: 192.168.10.1

elasticsearch.yml //master

// cluster.name: prod
// #
//# ------------------------------------ Node ------------------------------------
//#
// # Use a descriptive name for the node:
// #
// node.name: "nodetwo"
//node.master: true
// node.data: false
//#network.host: 192.168.10.150
// transport.host: 192.168.10.150
// #
// # Add custom attributes to the node:
// #
// #node.attr.rack: r1
// discovery.zen.ping.unicast.hosts: ["192.168.10.100", "192.168.10.150"]


(Jason Tedor) #8

From your first post:

Now you've changed the configuration, but you only changed the cluster name on one of the nodes (you have cluster.name: prod commented versus uncommented in the two configuration files. The nodes will not join if their cluster names do not match.

I think that you're making this harder than it is. Let's start two nodes, one that is master eligible, the other that is not. The master will bind to 192.168.1.232, the non-master node will bind to 192.168.1.218. Let's start with the simplest configuration that achieves this:

master node (on 192.168.1.232):

$ cat ~/elasticsearch/elasticsearch-5.3.0/config/elasticsearch.yml 
node.master: true
network.host: 192.168.1.232

non-master node (on 192.168.1.218):

$ cat ~/elasticsearch/elasticsearch-5.3.0/config/elasticsearch.yml 
node.master: false
network.host: 192.168.1.218
discovery.zen.ping.unicast.hosts: [ "192.168.1.232" ]

These nodes start fine, and the non-master node joins the master. From the non-master node:

[2017-04-18T08:30:36,829][INFO ][o.e.c.s.ClusterService   ] [j52qxt4] detected_master {IZnpRfC}{IZnpRfCWR-y9Fsz8HnOOxw}{iEUhEOSmTyKEfCe5EpTAuA}{192.168.1.232}{192.168.1.232:9300}, added {{IZnpRfC}{IZnpRfCWR-y9Fsz8HnOOxw}{iEUhEOSmTyKEfCe5EpTAuA}{192.168.1.232}{192.168.1.232:9300},}, reason: zen-disco-receive(from master [master {IZnpRfC}{IZnpRfCWR-y9Fsz8HnOOxw}{iEUhEOSmTyKEfCe5EpTAuA}{192.168.1.232}{192.168.1.232:9300} committed version [3]])

From the master node:

[2017-04-18T08:30:36,823][INFO ][o.e.c.s.ClusterService   ] [IZnpRfC] added {{j52qxt4}{j52qxt45R4SBYnlg1g_cIQ}{RGYLtoWWR7q-TT0RH9ddsA}{192.168.1.218}{192.168.1.218:9300},}, reason: zen-disco-node-join[{j52qxt4}{j52qxt45R4SBYnlg1g_cIQ}{RGYLtoWWR7q-TT0RH9ddsA}{192.168.1.218}{192.168.1.218:9300}]

From /_cat/nodes:

$ curl -XGET 192.168.1.232:9200/_cat/nodes
192.168.1.232 22  95 1 0.21 0.14 0.05 mdi * IZnpRfC
192.168.1.218 21 100 8 2.85           di  - j52qxt4

From here, you can add more configuration, taking one step at a time.


(mabrouk) #9

I am sorry but I still having problems

[root@nodetwo Desktop]# service elasticsearch status
elasticsearch (pid 4438) is running...
[root@nodetwo Desktop]# curl -XGET 'http://192.168.10.150:9200'
curl: (7) couldn't connect to host
[root@nodetwo Desktop]# curl -XGET 'http://nodetwo:9200'
curl: (7) couldn't connect to host
[root@nodetwo Desktop]# curl -XGET 'http://localhost:9200'
curl: (7) couldn't connect to host
[root@nodetwo Desktop]# vi /var/log/elasticsearch/hli.log
[root@nodetwo Desktop]# service elasticsearch status
elasticsearch (pid 4438) is running...
[root@nodetwo Desktop]# service elasticsearch status
elasticsearch dead but subsys locked


(Jason Tedor) #10

Please run jps, and lsof -p 4438 -P -n (or whatever the current pid is the next time that you execute service elasticsearch status) and share the output here and also show the logs from /var/log/elasticsearch.


(mabrouk) #11

thos is the output of lsof:

 [root@nodetwo Desktop]# jps
    6033 Jps
    5846 Elasticsearch
    [root@nodetwo Desktop]# lsof -p 5846 -P -n
    COMMAND  PID          USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
    java    5846 elasticsearch  cwd    DIR                8,2     4096 4460183 /usr/share/elasticsearch
    java    5846 elasticsearch  rtd    DIR                8,2     4096       2 /
    java    5846 elasticsearch  txt    REG                8,2     9488 4590617 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/bin/java
    java    5846 elasticsearch  mem    REG                8,2   161704 2228226 /lib64/ld-2.12.so
    java    5846 elasticsearch  mem    REG                8,2    23088 2228246 /lib64/libdl-2.12.so
    java    5846 elasticsearch  mem    REG                8,2  1930416 2228230 /lib64/libc-2.12.so
    java    5846 elasticsearch  mem    REG                8,2   146592 2228233 /lib64/libpthread-2.12.so
    java    5846 elasticsearch  mem    REG                8,2    47760 2228240 /lib64/librt-2.12.so
    java    5846 elasticsearch  mem    REG                8,2   600048 2228254 /lib64/libm-2.12.so
    java    5846 elasticsearch  mem    REG                8,2    91096 2228250 /lib64/libz.so.1.2.3
    java    5846 elasticsearch  mem    REG                8,2    93320 2228256 /lib64/libgcc_s-4.4.7-20120601.so.1
    java    5846 elasticsearch  mem    REG                8,2   989840 3284445 /usr/lib64/libstdc++.so.6.0.13
    java    5846 elasticsearch  mem    REG                8,2    60136 4590696 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/amd64/jli/libjli.so
    java    5846 elasticsearch  mem    REG                8,2 99174448 3296618 /usr/lib/locale/locale-archive
    java    5846 elasticsearch  DEL    REG                8,2          2621644 /tmp/jna--1985354563/jna2101214435842040982.tmp
    java    5846 elasticsearch  mem    REG                8,2    73400 4590648 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/amd64/libnio.so
    java    5846 elasticsearch  mem    REG                8,2    98272 4590647 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/amd64/libnet.so
    java    5846 elasticsearch  mem    REG                8,2    36008 4590645 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/amd64/libmanagement.so
    java    5846 elasticsearch  mem    REG                8,2   748363 4590697 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/jsse.jar
    java    5846 elasticsearch  mem    REG                8,2  2032041 4590678 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/ext/nashorn.jar
    java    5846 elasticsearch  mem    REG                8,2   794991 4460205 /usr/share/elasticsearch/lib/jts-1.13.jar
    java    5846 elasticsearch  mem    REG                8,2   632037 4460203 /usr/share/elasticsearch/lib/joda-time-2.9.5.jar
    java    5846 elasticsearch  mem    REG                8,2    78074 4460204 /usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar
    java    5846 elasticsearch  mem    REG                8,2    40540 4460201 /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.1.jar
    java    5846 elasticsearch  mem    REG                8,2   187507 4460219 /usr/share/elasticsearch/lib/lucene-sandbox-6.2.1.jar
    java    5846 elasticsearch  mem    REG                8,2   150045 4460214 /usr/share/elasticsearch/lib/lucene-join-6.2.1.jar
    java    5846 elasticsearch  mem    REG                8,2   246927 4460223 /usr/share/elasticsear> Preformatted text

(mabrouk) #12
ch/lib/lucene-suggest-6.2.1.jar
java    5846 elasticsearch  mem    REG                8,2    56583 4460220 /usr/share/elasticsearch/lib/lucene-spatial-6.2.1.jar
java    5846 elasticsearch  mem    REG                8,2   198736 4460207 /usr/share/elasticsearch/lib/log4j-api-2.6.2.jar
java    5846 elasticsearch  mem    REG                8,2  1472270 4460209 /usr/share/elasticsearch/lib/lucene-analyzers-common-6.2.1.jar
java    5846 elasticsearch  mem    REG                8,2   269295 4460225 /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
java    5846 elasticsearch  mem    REG                8,2  1191946 4460208 /usr/share/elasticsearch/lib/log4j-core-2.6.2.jar
java    5846 elasticsearch  mem    REG                8,2  2581808 4460211 /usr/share/elasticsearch/lib/lucene-core-6.2.1.jar
java    5846 elasticsearch  mem    REG                8,2  9437294 4460196 /usr/share/elasticsearch/lib/elasticsearch-5.0.2.jar
java    5846 elasticsearch  DEL    REG                8,2          2621671 /tmp/ffiWQMgGg
java    5846 elasticsearch  mem    REG                8,2  1137286 4460202 /usr/share/elasticsearch/lib/jna-4.2.2.jar
java    5846 elasticsearch  mem    REG                8,2   109723 4460195 /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
java    5846 elasticsearch  mem    REG                8,2   404072 4460218 /usr/share/elasticsearch/lib/lucene-queryparser-6.2.1.jar
java    5846 elasticsearch  mem    REG                8,2   187537 4460226 /usr/share/elasticsearch/lib/spatial4j-0.6.jar

(mabrouk) #13
>     java    5846 elasticsearch  mem    REG                8,2   280014 4460198 /usr/share/elasticsearch/lib/jackson-core-2.8.1.jar
>     java    5846 elasticsearch  mem    REG                8,2  1140290 4460197 /usr/share/elasticsearch/lib/hppc-0.7.1.jar
>     java    5846 elasticsearch  mem    REG                8,2 73060419 4590711 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/rt.jar
>     java    5846 elasticsearch  mem    REG                8,2     7419 4460224 /usr/share/elasticsearch/lib/securesm-1.1.jar
>     java    5846 elasticsearch  mem    REG                8,2    50196 4460199 /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.1.jar
>     java    5846 elasticsearch  mem    REG                8,2   196508 4460221 /usr/share/elasticsearch/lib/lucene-spatial-extras-6.2.1.jar
>     java    5846 elasticsearch  mem    REG                8,2   142514 4460216 /usr/share/elasticsearch/lib/lucene-misc-6.2.1.jar
>     java    5846 elasticsearch  mem    REG                8,2   225188 4460217 /usr/share/elasticsearch/lib/lucene-queries-6.2.1.jar
>     java    5846 elasticsearch  mem    REG                8,2    87409 4460210 /usr/share/elasticsearch/lib/lucene-backward-codecs-6.2.1.jar
>     java    5846 elasticsearch  mem    REG                8,2   144208 4460213 /usr/share/elasticsearch/lib/lucene-highlighter-6.2.1.jar
>     java    5846 elasticsearch  mem    REG                8,2    34152 4590655 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/amd64/libzip.so
>     java    5846 elasticsearch  mem    REG                8,2    66432 2228255 /lib64/libnss_files-2.12.so
>     java    5846 elasticsearch  mem    REG                8,2    49754 4460227 /usr/share/elasticsearch/lib/t-digest-3.0.jar

/usr/share/elasticsearch/lib/log4j-core-2.6.2.jar

>     java    5846 elasticsearch   19r   REG                8,2   269295 4460225 /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
>     java    5846 elasticsearch   20r   REG                8,2  1472270 4460209 /usr/share/elasticsearch/lib/lucene-analyzers-common-6.2.1.jar
>     java    5846 elasticsearch   21r   REG                8,2   198736 4460207 /usr/share/elasticsearch/lib/log4j-api-2.6.2.jar
>     java    5846 elasticsearch   22r   REG                8,2    56583 4460220 /usr/share/elasticsearch/lib/lucene-spatial-6.2.1.jar
>     java    5846 elasticsearch   23r   REG                8,2   246927 4460223 /usr/share/elasticsearch/lib/lucene-suggest-6.2.1.jar
>     java    5846 elasticsearch   24r   REG                8,2   150045 4460214 /usr/share/elasticsearch/lib/lucene-join-6.2.1.jar
>     java    5846 elasticsearch   25r   REG                8,2   187507 4460219 /usr/share/elasticsearch/lib/lucene-sandbox-6.2.1.jar
>     java    5846 elasticsearch   26r   REG                8,2    40540 4460201 /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.1.jar
>     java    5846 elasticsearch   27r   REG                8,2    78074 4460204 /usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar
>     java    5846 elasticsearch   28r   REG                8,2   404072 4460218 /usr/share/elasticsearch/lib/lucene-queryparser-6.2.1.jar
>     java    5846 elasticsearch   29r   REG                8,2    49754 4460227 /usr/share/elasticsearch/lib/t-digest-3.0.jar
>     java    5846 elasticsearch   30r   REG                8,2    50196 4460199 /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.1.jar
>     java    5846 elasticsearch   31r   REG                8,2     7419 4460224 /usr/share/elasticsearch/lib/securesm-1.1.jar
>     java    5846 elasticsearch   32r   REG                8,2   109723 4460195 /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
>     java    5846 elasticsearch   33r   REG                8,2   632037 4460203 /usr/share/elasticsearch/lib/joda-time-2.9.5.jar
>     java    5846 elasticsearch   34r   REG                8,2   794991 4460205 /usr/share/elasticsearch/lib/jts-1.13.jar
>     java    5846 elasticsearch   35r   REG                8,2  1137286 4460202 /usr/share/elasticsearch/lib/jna-4.2.2.jar
>     java    5846 elasticsearch   36r   REG                8,2  2032041 4590678 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/ext/nashorn.jar
>     java    5846 elasticsearch   37w   REG                8,2    22702 1441995 /var/log/elasticsearch/hli.log
>     java    5846 elasticsearch   38u  unix 0xffff88007b5c8840      0t0   39805 socket
>     java    5846 elasticsearch   39w   REG                8,2      804 1442662 /var/log/elasticsearch/hli_deprecation.log
>     java    5846 elasticsearch   40r   REG                8,2     3992 3017021 /etc/elasticsearch/log4j2.properties
>     java    5846 elasticsearch   41w   REG                8,2        0 1442688 /var/log/elasticsearch/hli_index_search_slowlog.log
>     java    5846 elasticsearch   42w   REG                8,2        0 1442689 /var/log/elasticsearch/hli_index_indexing_slowlog.log
>     java    5846 elasticsearch   43r   REG                8,2   748363 4590697 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-1.b13.el6.x86_64/jre/lib/jsse.jar
>     java    5846 elasticsearch   44r   CHR                1,8      0t0    4605 /dev/random
>     java    5846 elasticsearch   45r   CHR                1,9      0t0    4606 /dev/urandom
>     java    5846 elasticsearch   46r   CHR                1,8      0t0    4605 /dev/random
>     java    5846 elasticsearch   47r   CHR                1,8      0t0    4605 /dev/random
>     java    5846 elasticsearch   48r   CHR                1,9      0t0    4606 /dev/urandom
>     java    5846 elasticsearch   49r   CHR                1,9      0t0    4606 /dev/urandom
>     java    5846 elasticsearch   50u   REG                8,2     4096 2621671 /tmp/ffiWQMgGg (deleted)
>     java    5846 elasticsearch   51wW  REG                8,2        0 1442132 /var/lib/elasticsearch/nodes/0/node.lock

(mabrouk) #14
/var/log/elasticsearch/*.log
>     [2017-04-18T16:09:32,603][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
>     java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
>     	at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:361) ~[elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:158) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.2.jar:5.0.2]
[possibly], types [ext4]
>     [2017-04-18T16:09:56,912][INFO ][o.e.e.NodeEnvironment    ] [nodetwo] heap size [1.9gb], compressed ordinary object pointers [true]
>     [2017-04-18T16:09:58,324][INFO ][o.e.n.Node               ] [nodetwo] version[5.0.2], pid[5210], build[f6b4951/2016-11-24T10:07:18.101Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
>     [2017-04-18T16:26:35,480][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
>     java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
>     	at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:361) ~[elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:158) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.0.2.jar:5.0.2]
>     	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.0.2.jar:5.0.2]
>     [2017-04-18T16:27:33,352][INFO ][o.e.n.Node               ] [nodetwo] initializing ...
>     [2017-04-18T16:27:41,785][INFO ][o.e.e.NodeEnvironment    ] [nodetwo] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [67.6gb], net total_space [76.4gb], spins? [possibly], types [ext4]
>     [2017-04-18T16:27:41,786][INFO ][o.e.e.NodeEnvironment    ] [nodetwo] heap size [1.9gb], compressed ordinary object pointers [true]
>     [2017-04-18T16:27:42,108][INFO ][o.e.n.Node               ] [nodetwo] version[5.0.2], pid[5846], build[f6b4951/2016-11-24T10:07:18.101Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
>     [2017-04-18T16:29:57,945][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [aggs-matrix-stats]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [ingest-common]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [lang-expression]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [lang-groovy]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [lang-mustache]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [lang-painless]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [percolator]
>     [2017-04-18T16:29:58,088][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [reindex]
>     [2017-04-18T16:29:58,089][INFO ][o.e.p.PluginsService     ] [nodetwo] loaded module [transport-netty3]
>   
>     [2017-04-18T16:29:58,090][INFO ][o.e.p.Plugi     ] [nodetwo] no plugins loaded
>     [2017-04-18T16:31:13,845][INFO ][o.e.n.Node               ] [nodetwo] initialized
>     [2017-04-18T16:31:13,887][INFO ][o.e.n.Node               ] [nodetwo] starting ...
>     [2017-04-18T16:31:15,081][INFO ][o.e.t.TransportService   ] [nodetwo] publish_address {192.168.10.150:9300}, bound_addresses {192.168.10.150:9300}
>     [2017-04-18T16:31:15,091][INFO ][o.e.b.BootstrapCheck     ] [nodetwo] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
>     [2017-04-18T16:31:15,095][ERROR][o.e.b.Bootstrap          ] [nodetwo] node validation exception
>     bootstrap checks failed
>     max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
>     [2017-04-18T16:31:15,219][INFO ][o.e.n.Node               ] [nodetwo] stopping ...
>     [2017-04-18T16:31:15,389][INFO ][o.e.n.Node               ] [nodetwo] stopped
>     [2017-04-18T16:31:15,389][INFO ][o.e.n.Node               ] [nodetwo] closing ...
>     [2017-04-18T16:31:15,410][INFO ][o.e.n.Node               ] [nodetwo] closed

(Jason Tedor) #15

Right here:

[2017-04-18T16:31:15,091][INFO ][o.e.b.BootstrapCheck     ] [nodetwo] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-18T16:31:15,095][ERROR][o.e.b.Bootstrap          ] [nodetwo] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

Your node is failing on startup because your ulimit for nproc is too low.


(mabrouk) #16

What should I do ?
ulimit -u 2048


(Jason Tedor) #17

Typically that is not enough. You often have to make a change in /etc/security/limits.conf or /etc/security/limits.conf.d (and then there are obnoxious things like some OS that ship with defaults that override your attempts (look for a file like /etc/security/limits.d/90-nproc.conf or the sort to see if you're going to run into this)).


(mabrouk) #18

I corrected the previous error but elasticsearch does not start

[2017-04-19T11:41:38,466][INFO ][o.e.n.Node               ] [nodeone] starting ...
[2017-04-19T11:41:39,541][INFO ][o.e.t.TransportService   ] [nodeone] publish_address {192.168.10.100:9300}, bound_addresses {192.168.10.100:9300}
[2017-04-19T11:41:39,616][INFO ][o.e.b.BootstrapChecks    ] [nodeone] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-19T11:41:39,619][ERROR][o.e.b.Bootstrap          ] [nodeone] node validation exception
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-04-19T11:41:39,621][INFO ][o.e.n.Node               ] [nodeone] stopping ...
[2017-04-19T11:41:39,720][INFO ][o.e.n.Node               ] [nodeone] stopped
[2017-04-19T11:41:39,720][INFO ][o.e.n.Node               ] [nodeone] closing ...
[2017-04-19T11:41:40,338][INFO ][o.e.n.Node               ] [nodeone] closed

(Jason Tedor) #19

Where did you get stuck after reading the log messages, looking at the docs for the bootstrap checks, and taking the appropriate action to address?


(mabrouk) #20

Elasticsearch starts for some seconds and stop but the master it's work good