We had a Elasticsearch cluster when i check the nodes its is showing 3 nodes active but in the logs it is saying one node is not active

Elastic Stack Elasticsearch
1

we had a elasticsearch cluster with 3 nodes and all the nodes are running i checked that by using

[root@ip-10-0-0-170 elasticsearch]# curl --user elastic:yLwz4s2yRGiUQsXyFkXw  -X GET "10.0.0.170:9200/_cat/nodes?pretty"
10.0.0.158 43 57 1 0.66 0.66 0.95 dilm * ip-10-0-0-158
10.0.0.81  52 60 1 1.17 1.23 1.43 dilm - ip-10-0-0-81
10.0.0.170 28 96 1 2.13 2.21 2.09 dilm - ip-10-0-0-170

and the result of

[centos@ip-10-0-0-170 ~]$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-06-12 09:07:31 UTC; 25min ago
     Docs: http://www.elastic.co
 Main PID: 10738 (java)
   Memory: 1.5G
   CGroup: /system.slice/elasticsearch.service
           ├─10738 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=U...
           └─10842 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jun 12 09:07:14 ip-10-0-0-170.us-west-1.compute.internal systemd[1]: Starting Elasticsearch...
Jun 12 09:07:16 ip-10-0-0-170.us-west-1.compute.internal elasticsearch[10738]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be rem...e release.
Jun 12 09:07:31 ip-10-0-0-170.us-west-1.compute.internal systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.

but when i check the logs there is another scenario

[2020-06-12T09:07:40,645][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] not active
[2020-06-12T09:07:40,682][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T09:07:40,682][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] not active
[2020-06-12T09:07:40,732][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T09:07:40,733][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] not active

this is the same scenario in two of the nodes

[2020-06-12T00:00:00,197][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-81] not active
[2020-06-12T00:00:00,353][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-81] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
[2020-06-12T00:00:00,353][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-81] not active
[2020-06-12T00:00:00,376][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-81] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}

and my elsaticsearch.yml file is

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: argoid
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ip-10-0-0-170
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/1/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
bootstrap.system_call_filter: false
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.0.0.170
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["ip-10-0-0-170", "ip-10-0-0-81", "ip-10-0-0-158"]
cluster.initial_master_nodes: ["ip-10-0-0-170", "ip-10-0-0-81", "ip-10-0-0-158"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

#xpack.security.authc.realms.keycloak.order: 0
#xpack.security.authc.realms.keycloak.type: keycloak
#xpack.security.authc.realms.keycloak.config: keycloak.config
xpack.security.enabled: true
#xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE

can anyone tell me why it is happening

thanks in advance

2

You are looking at TRACE logs which are only really useful for deep debugging when read alongside the source code. The fix is therefore to remove these two lines from your config:

logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE
3

I had removed these two lines

logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
logger.org.elasticsearch.discovery: TRACE

now i'm seeing this in argoid.log

[2020-06-12T00:00:00,375][TRACE][o.e.d.PeerFinder         ] [ip-10-0-0-170] deactivating and setting leader to {ip-10-0-0-158}{M5FiOa2KT263Vlni_7_5xQ}{xG43a8MUQpOPFAUA4NRTDw}{10.0.0.158}{10.0.0.158:9300}{dilm}{ml.machine_memory=16656257024, ml.max_open_jobs=20, xpack.installed=true}
"argoid.log" 9281L, 1685581C
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]
4

Again, this is a TRACE log, you can ignore it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.