Un-Assigned Shards after the restart of the ES cluster on Kubernetes

pulir · February 7, 2022, 6:58am

I am using Elasticsearch 7.10.2 deployed on k8s cluster. The master and data nodes are backed by PVs for persisting the data. I have created a Cluster of 3 master and 3 data nodes, with 2 client nodes dedicated. My problem is , when I ingest some data into the data base and re-create the Elasticsearch Cluster. [ the Data , Master nodes are deleted and recreated [ pods ] , then the master and data nodes latches to the same PVs with which it has associated last time ]. After re-creating the cluster the SHARD are shown as UN-ASSIGNED, with the reason 'Node Left'.

So, can you clarify me , when we are preserving the data on the PVs. Why it is showing 'UN-ASSIGNED' even the data/master nodes latching to the same PVs. My worry is in k8s environment then how can we make the data stateful.

warkolm · February 7, 2022, 8:07am

Welcome to our community!

Please share your Elasticsearch logs from the master.

pulir · February 7, 2022, 12:05pm

Here are the logs from all the 3 master nodes:

master-0


Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:08,850][INFO ][o.e.n.Node               ] [elasticsearch-master-0] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node               ] [elasticsearch-master-0] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node               ] [elasticsearch-master-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9472843382414815717, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736,  -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:10,345][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-0] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [analysis-common]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [geo]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-common]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-geoip]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-user-agent]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [kibana]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-expression]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-mustache]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-painless]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [mapper-extras]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [parent-join]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [percolator]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [rank-eval]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [reindex]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [repository-url]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [transport-netty4]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [repository-s3]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-0] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [28.6gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-0] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:10,507][INFO ][o.e.n.Node               ] [elasticsearch-master-0] node name [elasticsearch-master-0], node ID [0NGeq5GrRD6gG6BdgKQH7A], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:14,365][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-0] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:14,440][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-0] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:14,826][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:15,061][INFO ][o.e.n.Node               ] [elasticsearch-master-0] initialized
[2022-02-07T11:56:15,062][INFO ][o.e.n.Node               ] [elasticsearch-master-0] starting ...
[2022-02-07T11:56:15,186][INFO ][o.e.t.TransportService   ] [elasticsearch-master-0] publish_address {10.233.90.62:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:15,266][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.96.102:52548}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,481][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.92.86:42818}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,497][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:15,501][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-0] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:19,124][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}, term: 5, version: 2463, reason: ApplyCommitRequest{term=5, version=2463, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,142][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-0] publish_address {10.233.90.62:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:19,143][INFO ][o.e.n.Node               ] [elasticsearch-master-0] started
[2022-02-07T11:56:19,344][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}, term: 5, version: 2465, reason: ApplyCommitRequest{term=5, version=2465, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,557][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}, term: 5, version: 2466, reason: ApplyCommitRequest{term=5, version=2466, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:23,358][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}, term: 5, version: 2467, reason: ApplyCommitRequest{term=5, version=2467, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,771][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: ApplyCommitRequest{term=5, version=2468, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,885][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: ApplyCommitRequest{term=5, version=2469, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:00,177][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: ApplyCommitRequest{term=5, version=2470, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}

system · February 7, 2022, 12:05pm

Opendistro is an AWS run product and differs from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

pulir · February 7, 2022, 12:07pm

master-1


Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:13,519][INFO ][o.e.n.Node               ] [elasticsearch-master-1] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:13,523][INFO ][o.e.n.Node               ] [elasticsearch-master-1] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:13,524][INFO ][o.e.n.Node               ] [elasticsearch-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9633479102114016921, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:14,968][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-1] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:14,969][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [analysis-common]
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [geo]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-common]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-geoip]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-user-agent]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [kibana]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-expression]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-mustache]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-painless]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [mapper-extras]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [parent-join]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [percolator]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [rank-eval]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [reindex]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [repository-url]
[2022-02-07T11:56:14,977][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [transport-netty4]
[2022-02-07T11:56:14,977][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [repository-s3]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:15,014][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [26.5gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:15,015][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:15,112][INFO ][o.e.n.Node               ] [elasticsearch-master-1] node name [elasticsearch-master-1], node ID [auRjv5lkT3iDaCtvM7_txA], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:18,670][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:18,744][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-1] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:19,084][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:19,280][INFO ][o.e.n.Node               ] [elasticsearch-master-1] initialized
[2022-02-07T11:56:19,281][INFO ][o.e.n.Node               ] [elasticsearch-master-1] starting ...
[2022-02-07T11:56:19,415][INFO ][o.e.t.TransportService   ] [elasticsearch-master-1] publish_address {10.233.105.88:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:19,521][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.105.88:9300, remoteAddress=/10.233.90.62:56514}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:19,685][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:19,688][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-1] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:19,981][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] elected-as-master ([2] nodes joined)[{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr} elect leader, {elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 5, version: 2463, delta: master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}
[2022-02-07T11:56:20,084][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}, term: 5, version: 2463, reason: Publication{term=5, version=2463}
[2022-02-07T11:56:20,141][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-1] publish_address {10.233.105.88:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:20,141][INFO ][o.e.n.Node               ] [elasticsearch-master-1] started
[2022-02-07T11:56:20,210][INFO ][o.e.g.GatewayService     ] [elasticsearch-master-1] recovered [6] indices into cluster_state
[2022-02-07T11:56:20,230][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [elasticsearch-master-1] get managed-index failed: NoShardAvailableActionException[No shard available for [org.elasticsearch.action.get.MultiGetShardRequest@e10609a]]
[2022-02-07T11:56:20,250][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir} join existing leader], term: 5, version: 2465, delta: added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}
[2022-02-07T11:56:20,271][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [elasticsearch-master-1] Failed to get ISM templates: Failed to execute phase [query], all shards failed
[2022-02-07T11:56:20,313][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}, term: 5, version: 2465, reason: Publication{term=5, version=2465}
[2022-02-07T11:56:20,481][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir} join existing leader], term: 5, version: 2466, delta: added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}
[2022-02-07T11:56:20,538][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}, term: 5, version: 2466, reason: Publication{term=5, version=2466}
[2022-02-07T11:56:24,288][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr} join existing leader], term: 5, version: 2467, delta: added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}
[2022-02-07T11:56:24,401][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}, term: 5, version: 2467, reason: Publication{term=5, version=2467}
[2022-02-07T11:57:00,696][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple} join existing leader], term: 5, version: 2468, delta: added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}
[2022-02-07T11:57:00,800][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: Publication{term=5, version=2468}
[2022-02-07T11:57:00,803][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple} join existing leader], term: 5, version: 2469, delta: added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}
[2022-02-07T11:57:01,103][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: Publication{term=5, version=2469}
[2022-02-07T11:57:01,106][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple} join existing leader], term: 5, version: 2470, delta: added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}
[2022-02-07T11:57:01,402][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: Publication{term=5, version=2470}
[2022-02-07T12:01:19,692][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-master-1] Running full sweep

system · February 7, 2022, 12:07pm

Opendistro is an AWS run product and differs from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

pulir · February 7, 2022, 12:07pm

master-2

Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:17,803][INFO ][o.e.n.Node               ] [elasticsearch-master-2] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:17,808][INFO ][o.e.n.Node               ] [elasticsearch-master-2] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:17,808][INFO ][o.e.n.Node               ] [elasticsearch-master-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-12968836566142102889, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:19,165][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-2] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:19,166][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:19,170][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:19,170][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [analysis-common]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [geo]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-common]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-geoip]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-user-agent]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [kibana]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-expression]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-mustache]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-painless]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [mapper-extras]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [parent-join]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [percolator]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [rank-eval]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [reindex]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [repository-url]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [transport-netty4]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [repository-s3]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:19,214][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-2] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [15.1gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:19,214][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-2] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:19,313][INFO ][o.e.n.Node               ] [elasticsearch-master-2] node name [elasticsearch-master-2], node ID [GaxnS852Q2yH78QyfsscBA], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:22,951][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-2] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:23,027][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-2] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:23,364][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:23,620][INFO ][o.e.n.Node               ] [elasticsearch-master-2] initialized
[2022-02-07T11:56:23,620][INFO ][o.e.n.Node               ] [elasticsearch-master-2] starting ...
[2022-02-07T11:56:23,747][INFO ][o.e.t.TransportService   ] [elasticsearch-master-2] publish_address {10.233.92.87:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:24,032][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:24,035][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-2] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:24,336][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir},{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr},{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir},{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}, term: 5, version: 2467, reason: ApplyCommitRequest{term=5, version=2467, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:24,387][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-2] publish_address {10.233.92.87:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:24,387][INFO ][o.e.n.Node               ] [elasticsearch-master-2] started
[2022-02-07T11:57:00,703][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: ApplyCommitRequest{term=5, version=2468, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:00,802][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: ApplyCommitRequest{term=5, version=2469, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:01,107][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: ApplyCommitRequest{term=5, version=2470, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T12:01:24,038][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-master-2] Running full sweep

Christian_Dahlqvist · February 7, 2022, 4:23pm

Do the shards eventually get assigned or do they stay unassigned indefinitely?

pulir · February 7, 2022, 5:08pm

They stay unassigned indefinitely.

you can see the following cluster state output, where the 'node-left' UUID jzQ0MPkmT jiJnNGCwwpo2g is differnt than the current node UUIDs 3ZY8c-4ZSQyohb7aqqCKyg,
q3gYQ8pfQoeQhl6tuzeBog , f-iHhwkYS9-oNeahbF9tAA . I could see that after the node/pod restart of the ES cluster. Each node/pod restarted is getting a new UUID, but the shard state was still with the old UUID. I hope this is the problem for the shards to remain UN-ASSIGNED state for ever. If so, is there a way to re-use the old UUID or we can configure some name for each of the data node to use it always the same name even after the restart.

system · March 7, 2022, 5:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.