Hello All
I have a cluster which was running in 7.7.0 version without issues and I just update it to 7.12.1 version and when I scale out the cluster the new instance is able to join the cluster but after a minute shows the following error and is removed from the cluster, once I restart the elasticsearch service the instance can join the cluster again without issues.
Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
Those are the logs:
[2021-05-06T02:25:47,525][INFO ][o.e.e.NodeEnvironment ] [i-XXXX] heap size [31gb], compressed ordinary object pointers [true]
[2021-05-06T02:25:47,545][INFO ][o.e.n.Node ] [i-XXXXXX] node name [i-XXXX], node ID [AIGno69SRkm7zWmUkCl3-w], cluster name [antik], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-05-06T02:25:51,367][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [i-XXXX] [controller/3626] [Main.cc@117] controller (64 bit): Version 7.12.1 (Build 4172997de5701c) Copyright (c) 2021 Elasticsearch BV
[2021-05-06T02:25:51,812][INFO ][o.e.x.s.a.s.FileRolesStore] [i-XXXXX] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-05-06T02:25:53,694][INFO ][o.e.t.NettyAllocator ] [i-XXXX] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2021-05-06T02:25:53,763][INFO ][o.e.d.DiscoveryModule ] [i-XXXXX] using discovery type [zen] and seed hosts providers [settings, ec2]
[2021-05-06T02:25:54,254][INFO ][o.e.g.DanglingIndicesState] [i-XXXX] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-05-06T02:25:54,846][INFO ][o.e.n.Node ] [i-XXXX] initialized
[2021-05-06T02:25:54,847][INFO ][o.e.n.Node ] [i-XXXX] starting ...
[2021-05-06T02:25:54,858][INFO ][o.e.x.s.c.f.PersistentCache] [i-XXXX] persistent cache index loaded
[2021-05-06T02:25:54,973][INFO ][o.e.t.TransportService ] [i-XXXX] publish_address {10.252.0.48:9300}, bound_addresses {0.0.0.0:9300}
[2021-05-06T02:25:55,093][INFO ][o.e.b.BootstrapChecks ] [i-XXXX] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-05-06T02:25:57,729][INFO ][o.e.c.s.ClusterApplierService] [i-XXXX] master node changed {previous [], current [{i-XXXXXX}{RCosKwrJQye0JKJfA2vcqw}{CH-jxA1zRYe1p7aGKW3pRA}{10.252.6.176}{10.252.6.176:9300}{cdfhilmrstw}{ml.machine_memory=66714148864, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=33287569408, transform.node=true}]}, added {{i-XXXXXXX}{zPbfqKpoTOWOzRAdY-ahOQ}{l18QLJJ_RwmBF1-4FafcPg}{10.252.3.77}{10.252.3.77:9300}{cdfhilmrstw}{ml.machine_memory=66714148864, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=33287569408, transform.node=true},{i-034e521477e085f89}{xwic8Zp_QU6MU2vSl_hRwA}{KTIemnXXRSWDVL5TROzaDA}{10.252.0.188}{10.252.0.188:9300}{cdfhilmrstw}{ml.machine_memory=66714148864, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=33287569408, transform.node=true},{i-XXXXX}{RCosKwrJQye0JKJfA2vcqw}{CH-jxA1zRYe1p7aGKW3pRA}{10.252.6.176}{10.252.6.176:9300}{cdfhilmrstw}{ml.machine_memory=66714148864, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=33287569408, transform.node=true}}, term: 22, version: 13438851, reason: ApplyCommitRequest{term=22, version=13438851, sourceNode={i-XXXXX}{RCosKwrJQye0JKJfA2vcqw}{CH-jxA1zRYe1p7aGKW3pRA}{10.252.6.176}{10.252.6.176:9300}{cdfhilmrstw}{ml.machine_memory=66714148864, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=33287569408, transform.node=true}}
[2021-05-06T02:25:57,899][INFO ][o.e.x.s.a.TokenService ] [i-XXXXX] refresh keys
[2021-05-06T02:25:58,146][INFO ][o.e.x.s.a.TokenService ] [i-XXXXXX] refreshed keys
[2021-05-06T02:25:58,188][INFO ][o.e.l.LicenseService ] [i-XXXXX] license [df5a6744-5863-44f8-a51e-310ab315f887] mode [basic] - valid
[2021-05-06T02:25:58,189][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [i-XXXX] Active license is now [BASIC]; Security is enabled
[2021-05-06T02:25:58,209][INFO ][o.e.h.AbstractHttpServerTransport] [i-XXXXXX] publish_address {10.252.0.48:9200}, bound_addresses {0.0.0.0:9200}
[2021-05-06T02:25:58,209][INFO ][o.e.n.Node ] [i-XXXXXX] started
[2021-05-06T02:25:58,404][INFO ][o.e.x.s.a.AuthenticationService] [i-XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2021-05-06T02:25:58,542][INFO ][o.e.x.s.a.AuthenticationService] [i-XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]```