# Unstable after starting nodes
**URL:** https://discuss.elastic.co/t/unstable-after-starting-nodes/200526
**Category:** Elasticsearch
**Created:** [September 21, 2019, 11:52am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526 "2019-09-21T11:52:22Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 21, 2019, 11:52am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/1 "2019-09-21T11:52:22Z")
Hi,
ES: v7.3.2
I create a brand new cluster with 3 master nodes, no problem all running fine. but each time that i want to add a node ( no master ) after some time ( 1h ) the node disappear, monitoring show node was removed, and the cluster is very slow and unstable. then i stop the node and everything come back fine.
I try this with several other nodes in different vlan , same result.
Is there a trick in the discovery part ?
Here's my config of master , for nodes i just put node.master: false
cluster.name: blabla
node.name: blabla-1
path.data: /opt/elasticsearch/data
path.logs: /var/log/elasticsearch
#bootstrap.memory\_lock: true
bootstrap.system\_call\_filter: false
network.host: _site_,_local_
http.port: 9200
discovery.seed\_hosts: ["blabla-1", "blabla-2","blabla-3"]
cluster.initial\_master\_nodes: ["blabla-1", "blabla-2","blabla-3"]
node.master: true
node.data: true
error it give when i add a node is :
master node changed {previous [{blabla-2}{XCPed0npS5m2Sub0AfqTQw}{38O\_1NF2RWSFi8
lB0eV3QA}{10.30.172.196}{10.30.172.196:9300}{dim}{ml.machine\_memory=67368509440, ml.max\_open\_jobs=20, xpack.installed=true}], current }, term: 5, version: 1182, reason: becoming candidate: onLeaderFailure
Caused by: org.elasticsearch.transport.RemoteTransportException: [blabla-2][10.30.172.196:9300][internal:coordination/fault\_detection/leader\_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: leader check from unknown node
---
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 21, 2019, 3:14pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/2 "2019-09-21T15:14:08Z")
For discovery.seed\_hosts: , do i have to enter all nodes including master and no master also ?
---
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 22, 2019, 8:35am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/3 "2019-09-22T08:35:40Z")
No one on 7.3.2 ?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 22, 2019, 9:32am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/4 "2019-09-22T09:32:05Z")
> [@Tex](#):
>
> `leader check from unknown node`
This means the node was already removed from the cluster. There will an earlier message saying why.
> [@Tex](#):
>
> For discovery.seed\_hosts: , do i have to enter all nodes including master and no master also ?
No, that setting should only mention master-eligible nodes. It doesn't sound like a discovery problem, however, because the node must have joined the cluster to get the message you are seeing.
---
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 22, 2019, 10:03am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/5 "2019-09-22T10:03:54Z")
ok tx.
I try several setup and try to put also all nodes as master but still the same problem.
after around 2h all monitoinr failed on every nodes, kibana graf show a cut on every nodes
log says
627x145](upload://84qRbupyNUbijgR66AMIdAqmlCr.png) [2019-09-22T11:40:59,739][DEBUG][o.e.a.a.c.s.TransportClusterStateAction] [blabla-1] no known master node, scheduling a retry
[2019-09-22T11:40:59,747][DEBUG][o.e.a.a.c.s.TransportClusterStateAction] [blabla-1] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2019-09-22T11:40:59,747][WARN][r.suppressed] [blabla-1] path: /\_cluster/settings, params: {include\_defaults=true}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
---
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 22, 2019, 10:05am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/6 "2019-09-22T10:05:23Z")
i've exactly the same setup in test and it runs since 2 months without any errors.
---
### Author:  [@Tex](https://discuss.elastic.co/u/Tex)
#### Post date: [September 22, 2019, 10:06am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/7 "2019-09-22T10:06:23Z")
everytime it take 1h30 - 2h and all failed. i'm lost.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 22, 2019, 11:02am UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/8 "2019-09-22T11:02:01Z")
> [@Tex](#):
>
> `org.elasticsearch.discovery.MasterNotDiscoveredException`
This node can't find the elected master node. I expect there are other log messages saying why.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [September 26, 2019, 12:10pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/9 "2019-09-26T12:10:14Z")
Have just updated to 7.3.2 from 6.8 and having the same issue
No any problems with hardware or network, but node periodically lost from cluster
In logs:
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-02][10.1.3.112:9300][internal:coordination/fault\_detection/leader\_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: leader check from unknown node
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 26, 2019, 12:13pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/10 "2019-09-26T12:13:36Z")
As I said above, `leader check from unknown node` means the node was already removed from the cluster and there will be an earlier message saying why. Look for the string `node-left`.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [September 26, 2019, 1:30pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/12 "2019-09-26T13:30:28Z")
Grep logs for "node-left":
>
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 26, 2019, 1:40pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/13 "2019-09-26T13:40:44Z")
Thanks. The first one is the one we're after:
`[2019-09-26T11:55:27,652][INFO][o.e.c.s.MasterService] [node-02] node-left[{node-00}{A9AdXLb5QA-ZMcicCn26OQ}{S9uJF_S1QNC1SsgTqmY3bQ}{10.1.3.110}{10.1.3.110:9300}{dim}{ml.machine_memory=134928560128, ml.max_open_jobs=20, xpack.installed=true} lagging], term: 3, version: 2093, reason: removed {{node-00}{A9AdXLb5QA-ZMcicCn26OQ}{S9uJF_S1QNC1SsgTqmY3bQ}{10.1.3.110}{10.1.3.110:9300}{dim}{ml.machine_memory=134928560128, ml.max_open_jobs=20, xpack.installed=true},}`
Note the `lagging` - this means this node took more than 2 minutes to process a cluster state update, which Elasticsearch 7.x treats as a failure of this node.
The logging around this is improved in 7.4 (and even more in 7.5) but in 7.3 I suggest you add
```auto
logger.org.elasticsearch.gateway.MetaStateService: TRACE
logger.org.elasticsearch.cluster.service: TRACE
```
This will give much more detail about why it is processing cluster states so slowly.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [September 26, 2019, 3:08pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/14 "2019-09-26T15:08:27Z")
>
This log is not from TRACE but contains `reason`
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [October 17, 2019, 3:03pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/15 "2019-10-17T15:03:51Z")
We have exactly the same issue (7.3.1) where get this error: [https://github.com/elastic/elasticsearch/blob/v7.3.1/server/src/main/java/org/elasticsearch/cluster/coordination/LeaderChecker.java#L180](https://github.com/elastic/elasticsearch/blob/v7.3.1/server/src/main/java/org/elasticsearch/cluster/coordination/LeaderChecker.java#L180).
We see a lot of timeouts too, but don't really know where they come from. The load isn't that extraordinary.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 17, 2019, 3:11pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/16 "2019-10-17T15:11:01Z")
Repeating [my last comment](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/13):
> The logging around this is improved in 7.4 (and even more in 7.5) but in 7.3 I suggest you add
>
> ```auto
> logger.org.elasticsearch.gateway.MetaStateService: TRACE
> logger.org.elasticsearch.cluster.service: TRACE
>
> ```
>
> This will give much more detail about why it is processing cluster states so slowly.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 14, 2019, 3:11pm UTC](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/17 "2019-11-14T15:11:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.