# Node discovery issue post upgrade to newer version of ELK stack
**URL:** https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741
**Category:** Elasticsearch
**Created:** [August 21, 2017, 9:56am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741 "2017-08-21T09:56:25Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 21, 2017, 9:56am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/1 "2017-08-21T09:56:26Z")
I have migrated all my ELK nodes to new versions. I have 2 ES nodes with configs as below
10.\*\*\*.235
`elk_es_crt_00`
`cluster.name: csm_elk_es_crt_00`
`node.name: "elkescrt00.***.net"`
`bootstrap.memory_lock: false`
`network.host: 192.168.0.1`
`transport.host: 127.0.0.1`
`http.port: 9200`
`http.host: 0.0.0.0`
`discovery.zen.ping.unicast.hosts: ["elkescrt00. ***.net", "elkescrt01.***.net"]`
10.\*\*\*.234
`elk_es_crt_01`
`cluster.name: csm_elk_es_crt_01`
`node.name: "elkescrt01.***.net"`
`bootstrap.memory_lock: false`
`network.host: 192.168.0.1`
`transport.host: 127.0.0.1`
`http.port: 9200`
`http.host: 0.0.0.0`
`discovery.zen.ping.unicast.hosts: ["elkescrt00. ***.net", "elkescrt01.***.net"]`
and my kibana.yml has below config;s
kibana.yml
`server.port: 5601`
`server.host: "10.***.232"`
`elasticsearch.url: "http://elkescrt01.***.net:9200"`
after migration ES indices in elk\_es\_crt\_01 has been hashed but not elk\_es\_crt\_00.
am i missing something?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 21, 2017, 10:07am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/2 "2017-08-21T10:07:06Z")
The configuration does not make sense to me. It looks like the nodes reside in different IP addresses and based on `discovery.zen.ping.unicast.hosts` it also look like you are expecting the nodes to form a cluster. This will however not be possible as the two nodes do not have the same `cluster.name` and `transport.host` only binds to localhost. What are you expecting the setup to look like? How did it work before the migration? Which version of Elasticsearch are you using?
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 21, 2017, 10:44am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/3 "2017-08-21T10:44:34Z")
Yes both nodes reside in different IP.
This was a cluster with 2 ES nodes before migration. you are correct am trying to form a cluster as before.
My ES version is 5.5.1.
Now i did bind the transport.host with localhost.
when i do `http://elkcrt.***.com:9200/_cat/nodes`
its showing only
`127.0.0.1 21 97 7 0.11 0.17 0.12 mdi * elkescrt00.***.net`
and `GET /_cat/nodes`
`127.0.0.1 11 96 1 0.00 0.11 0.20 mdi * elkescrt01.***.net`
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 21, 2017, 10:50am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/4 "2017-08-21T10:50:21Z")
In order to form a cluster the nodes need to:
1. Have the same cluster name
2. Be able to communicate via the transport protocol on port 9300
In your config the cluster name is different and the nodes can not reach each others transport port as it binds to localhost. You will need to change this in order for your cluster to form a cluster.
You should also set `discovery.zen.minimum_master_nodes` to 2 once you establish a cluster if both your nodes are master eligible.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 22, 2017, 10:10am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/5 "2017-08-22T10:10:06Z")
So should i mention the port number 9300 explicitly?
i changed the cluster names to be same and i think it was success and in logs i got the below
```
[o.e.c.s.ClusterService] [elkescrt00.***.net] new_master {elkescrt00.***.net}{qgbCeUyrQtSkrL9ciZcnlQ}{6o3tCU2tRg6QV95dnJrwBA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-22T04:55:17,735][INFO][o.e.h.n.Netty4HttpServerTransport] [elkescrt00.***.net] publish_address {10.***.235:9200}, bound_addresses {10.***.235:9200}
```
But the problem is when i check the nodes using
`http://elkcrt.***.com:9200/_cat/nodes`
it is giving me
`127.0.0.1 72 98 8 0.42 0.99 0.99 mdi * elkescrt01.***.net`
only 1 node.
I don't understand.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 22, 2017, 10:17am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/6 "2017-08-22T10:17:51Z")
Your node is still binding to `127.0.0.1`, which will not work. You need to bind to an IP address (`10. ***.234` and `10.***.235`) that can be reached from the other node.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 22, 2017, 10:54am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/7 "2017-08-22T10:54:52Z")
Thanks. So it should be
`network.host: 10.***.235` ?
If so am getting the error
`elasticsearch dead but subsys locked`
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 10:12am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/8 "2017-08-23T10:12:33Z")
Have been getting this error.
Not able to validate what it is.
I changed my es config to bind host IP with transport port but es service dies everytime.
```
10.***.235
elk_es_crt_00
cluster.name: csm_elk_es_crt_01
node.name: "elkescrt00.***.net"
bootstrap.memory_lock: true
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elkescrt00. ***.net", "elkescrt01.***.net"]
10.***.234
elk_es_crt_01
cluster.name: csm_elk_es_crt_01
node.name: "elkescrt01.***.net"
bootstrap.memory_lock: true
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elkescrt00. ***.net", "elkescrt01.***.net"]
```
Error in logs
```
[2017-08-23T05:07:17,732][INFO][o.e.t.TransportService] [elkescrt00.***.net] publish_address {10.162.119.235:9300}, bound_addresses {10.162.119.235:9300}
[2017-08-23T05:07:17,746][INFO][o.e.b.BootstrapChecks] [elkescrt00.***.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-23T05:07:17,748][ERROR][o.e.b.Bootstrap] [elkescrt00.***.net] node validation exception
[2] bootstrap checks failed
[1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-08-23T05:07:17,751][INFO][o.e.n.Node] [elkescrt00.***.net] stopping ...
[2017-08-23T05:07:17,797][INFO][o.e.n.Node] [elkescrt00.***.net] stopped
[2017-08-23T05:07:17,797][INFO][o.e.n.Node] [elkescrt00.***.net] closing ...
[2017-08-23T05:07:17,809][INFO][o.e.n.Node] [elkescrt00.***.net] closed
```
i really need to fix this soon.
Can somebody help.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 23, 2017, 11:29am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/9 "2017-08-23T11:29:56Z")
In addition to what you have here, you need to configure [network.host](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/important-settings.html#network.host) to a non-loopback interface, e.g. the IP of the host. Once you do this, Elasticsearch will determine that you are no longer running a single instance locally for development, and will add additional checks to ensure the deployment is sound and follow best practices. There is [system configuration](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/system-config.html#system-config) you will need to pay attention to and also a number of [bootstrap checks](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/bootstrap-checks.html#bootstrap-checks) that will need to pass. Information about any checks that fail will be written to the log.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 11:39am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/10 "2017-08-23T11:39:41Z")
Thanks for the quick reply.
I have same indices in both nodes and having same logs writing on both nodes.
How do i know i have a cluster for my 2 node ES set up?
Is this the command to know number of ES nodes in my cluster?
`GET /_nodes`
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 23, 2017, 11:42am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/11 "2017-08-23T11:42:46Z")
The [cat nodes API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html) will show the nodes in the cluster.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 11:54am UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/12 "2017-08-23T11:54:59Z")
> [@Christian\_Dahlqvist](#):
>
> non-loopback interface
the problem is as soon as i add non-loopback interface for ex
`network.host: 10.162.119.235`
and restart the ES service it gives me
`elasticsearch dead but subsys locked`
error.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 23, 2017, 12:07pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/13 "2017-08-23T12:07:10Z")
What is in the Elasticsearch logs?
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 12:20pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/14 "2017-08-23T12:20:38Z")
Here is the logs from 1 of the 2 nodes[elkescrt01.\*\*\*.net,elkescrt00.\*\*\*.net]. Its the same log except for the host name elkescrt01.\*\*\*.net .
```
[2017-08-23T06:19:06,719][INFO][o.e.n.Node] [elkescrt01.***.net] initializing ...
[2017-08-23T06:19:06,988][INFO][o.e.e.NodeEnvironment] [elkescrt01.***.net] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/elasticsearch-elasticsearch--data)]], net usable_space [1003gb], net total_space [1.1tb], spins? [possibly], types [ext4]
[2017-08-23T06:19:06,988][INFO][o.e.e.NodeEnvironment] [elkescrt01.***.net] heap size [5.9gb], compressed ordinary object pointers [true]
[2017-08-23T06:19:17,348][INFO][o.e.n.Node] [elkescrt01.***.net] node name [elkescrt01.***.net], node ID [Gn5tyVUAQKmetNeJGyjSFg]
[2017-08-23T06:19:17,348][INFO][o.e.n.Node] [elkescrt01.***.net] version[5.5.1], pid[5210], build[19c13d0/2017-07-18T20:44:24.823Z], OS[Linux/2.6.32-573.18.1.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_71/25.71-b15]
[2017-08-23T06:19:17,354][INFO][o.e.n.Node] [elkescrt01.***.net] JVM arguments [-Xms6g, -Xmx6g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-08-23T06:19:19,299][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [aggs-matrix-stats]
[2017-08-23T06:19:19,299][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [ingest-common]
[2017-08-23T06:19:19,299][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [lang-expression]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [lang-groovy]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [lang-mustache]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [lang-painless]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [parent-join]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [percolator]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [reindex]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [transport-netty3]
[2017-08-23T06:19:19,300][INFO][o.e.p.PluginsService] [elkescrt01.***.net] loaded module [transport-netty4]
[2017-08-23T06:19:19,301][INFO][o.e.p.PluginsService] [elkescrt01.***.net] no plugins loaded
[2017-08-23T06:19:22,653][INFO][o.e.d.DiscoveryModule] [elkescrt01.***.net] using discovery type [zen]
[2017-08-23T06:19:27,048][INFO][o.e.n.Node] [elkescrt01.***.net] initialized
[2017-08-23T06:19:27,048][INFO][o.e.n.Node] [elkescrt01.***.net] starting ...
[2017-08-23T07:16:43,879][INFO][o.e.t.TransportService] [elkescrt01.***.net] publish_address {10.162.119.234:9300}, bound_addresses {10.162.119.234:9300}
[2017-08-23T07:16:43,908][INFO][o.e.b.BootstrapChecks] [elkescrt01.***.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-23T07:16:43,911][ERROR][o.e.b.Bootstrap] [elkescrt01.***.net] node validation exception
[1] bootstrap checks failed
[1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-08-23T07:16:43,915][INFO][o.e.n.Node] [elkescrt01.***.net] stopping ...
[2017-08-23T07:16:43,974][INFO][o.e.n.Node] [elkescrt01.***.net] stopped
[2017-08-23T07:16:43,974][INFO][o.e.n.Node] [elkescrt01.***.net] closing ...
[2017-08-23T07:16:43,991][INFO][o.e.n.Node] [elkescrt01.***.net] closed
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 23, 2017, 12:23pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/15 "2017-08-23T12:23:49Z")
> [@Vivek\_Samaga](#):
>
> [2017-08-23T07:16:43,911][ERROR][o.e.b.Bootstrap] [elkescrt01.\*\*\*.net] node validation exception
> [1] bootstrap checks failed
> [1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
As you can see from the logs, the bootstrap checks are not passing, preventing the node from starting up. Go through the full [list of bootstrap checks](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/bootstrap-checks.html) and make sure they are all correctly configured.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 12:28pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/16 "2017-08-23T12:28:10Z")
> [@Christian\_Dahlqvist](#):
>
> node validation exception
I have done this @Christian_Dahlqvist and did set the ulimit -u to 2048 as root user.
But this message keeps popping.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 23, 2017, 12:39pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/17 "2017-08-23T12:39:13Z")
If you have installed from a package, Elasticsearch should be running as the `elasticsearch` user, not root.
---
### Author:  [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)
#### Post date: [August 23, 2017, 1:52pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/18 "2017-08-23T13:52:43Z")
can't thank enough 🙂 😃
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 20, 2017, 1:52pm UTC](https://discuss.elastic.co/t/node-discovery-issue-post-upgrade-to-newer-version-of-elk-stack/97741/19 "2017-09-20T13:52:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.