# We had a Elasticsearch cluster when i check the nodes its is showing 3 nodes active but in the logs it is saying one node is not active
**URL:** https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862
**Category:** Elasticsearch
**Created:** [June 12, 2020, 9:39am UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862 "2020-06-12T09:39:37Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@hello5346](https://discuss.elastic.co/u/hello5346)
#### Post date: [June 12, 2020, 9:39am UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862/1 "2020-06-12T09:39:37Z")
we had a elasticsearch cluster with 3 nodes and all the nodes are running i checked that by using
```
[root@ip-10-0-0-170 elasticsearch]# curl --user elastic:yLwz4s2yRGiUQsXyFkXw -X GET "10.0.0.170:9200/_cat/nodes?pretty"
10.0.0.158 43 57 1 0.66 0.66 0.95 dilm * ip-10-0-0-158
10.0.0.81 52 60 1 1.17 1.23 1.43 dilm - ip-10-0-0-81
10.0.0.170 28 96 1 2.13 2.21 2.09 dilm - ip-10-0-0-170
```
and the result of
```
[centos@ip-10-0-0-170 ~]$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-06-12 09:07:31 UTC; 25min ago
Docs: http://www.elastic.co
Main PID: 10738 (java)
Memory: 1.5G
CGroup: /system.slice/elasticsearch.service
├─10738 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=U...
└─10842 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Jun 12 09:07:14 ip-10-0-0-170.us-west-1.compute.internal systemd[1]: Starting Elasticsearch...
Jun 12 09:07:16 ip-10-0-0-170.us-west-1.compute.internal elasticsearch[10738]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be rem...e release.
Jun 12 09:07:31 ip-10-0-0-170.us-west-1.compute.internal systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.
```
but when i check the logs there is another scenario
```
[2020-06-12T09:07:40,645][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] not active
[2020-06-12T09:07:40,682][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T09:07:40,682][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] not active
[2020-06-12T09:07:40,732][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T09:07:40,733][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] not active
```
this is the same scenario in two of the nodes
```
[2020-06-12T00:00:00,197][TRACE][o.e.d.PeerFinder] [ip-10-0-0-81] not active
[2020-06-12T00:00:00,353][TRACE][o.e.d.PeerFinder] [ip-10-0-0-81] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T00:00:00,353][TRACE][o.e.d.PeerFinder] [ip-10-0-0-81] not active
[2020-06-12T00:00:00,376][TRACE][o.e.d.PeerFinder] [ip-10-0-0-81] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
```
and my elsaticsearch.yml file is
```
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: argoid
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ip-10-0-0-170
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/1/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
bootstrap.system_call_filter: false
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.0.0.170
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["ip-10-0-0-170", "ip-10-0-0-81", "ip-10-0-0-158"]
cluster.initial_master_nodes: ["ip-10-0-0-170", "ip-10-0-0-81", "ip-10-0-0-158"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#xpack.security.authc.realms.keycloak.order: 0
#xpack.security.authc.realms.keycloak.type: keycloak
#xpack.security.authc.realms.keycloak.config: keycloak.config
xpack.security.enabled: true
#xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE
```
can anyone tell me why it is happening
thanks in advance
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 12, 2020, 11:15am UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862/2 "2020-06-12T11:15:05Z")
You are looking at `TRACE` logs which are only really useful for deep debugging when read alongside the source code. The fix is therefore to remove these two lines from your config:
```auto
logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE
```
---
### Author:  [@hello5346](https://discuss.elastic.co/u/hello5346)
#### Post date: [June 12, 2020, 11:38am UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862/3 "2020-06-12T11:38:58Z")
I had removed these two lines
```
logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE
```
now i'm seeing this in argoid.log
```
[2020-06-12T00:00:00,375][TRACE][o.e.d.PeerFinder] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
"argoid.log" 9281L, 1685581C
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 12, 2020, 12:05pm UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862/4 "2020-06-12T12:05:13Z")
Again, this is a `TRACE` log, you can ignore it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 10, 2020, 12:05pm UTC](https://discuss.elastic.co/t/we-had-a-elasticsearch-cluster-when-i-check-the-nodes-its-is-showing-3-nodes-active-but-in-the-logs-it-is-saying-one-node-is-not-active/236862/5 "2020-07-10T12:05:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.