I have a k3s kubernetes cluster and an elasticsearch cluster deployment. I am trying to install minhash plugin and playing around with the plugin.
everything is normal and I can restore some old snapshots successfuly in es cluster and also installing of minhash plugin is successful.
then I put an index using this command
curl -XPUT -H "Content-Type: application/json" '192.168.120.53:9200/my_index2' '"index":{"analysis":{"analyzer":{"minhash_analyzer":{"type":"custom","tokenizer":"standard","filter":["minhash"]}}}}}'
then my_index2 is created successfuly. as soon as i try below command master nodes fail and the container restarts:
curl -XPUT -H "Content-Type: application/json" "192.168.120.53:9200/my_index2/my_type/_mapping?include_type_name=true" -d '{"my_type":{"properties":{"message":{"type":"string","copy_to":"minhash_value"},"minhash_value":{"type":"minhash","minhash_analyzer":"minhash_analyzer"}}}}'
below is my elasticsearch log:
[2021-05-04T14:30:04,457][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-1] publish_address {10.42.3.76:9200}, bound_addresses {[::]:9200}
[2021-05-04T14:30:04,457][INFO ][o.e.n.Node ] [elasticsearch-master-1] started
[2021-05-04T14:38:55,327][INFO ][o.e.c.c.Coordinator ] [elasticsearch-master-1] master node [{elasticsearch-master-2}{PTtLrgqkSzSeWh0OfJMpTg}{Ung6LwKlTNSRY4lgnndmTA}{10.42.3.74}{10.42.3.74:9300}{mr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [elasticsearch-master-2][10.42.3.74:9300][disconnected] disconnected
[2021-05-04T14:38:55,337][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] master node changed {previous [{elasticsearch-master-2}{PTtLrgqkSzSeWh0OfJMpTg}{Ung6LwKlTNSRY4lgnndmTA}{10.42.3.74}{10.42.3.74:9300}{mr}], current []}, term: 30, version: 180, reason: becoming candidate: onLeaderFailure
[2021-05-04T14:38:55,357][WARN ][o.e.c.NodeConnectionsService] [elasticsearch-master-1] failed to connect to {elasticsearch-master-2}{PTtLrgqkSzSeWh0OfJMpTg}{Ung6LwKlTNSRY4lgnndmTA}{10.42.3.74}{10.42.3.74:9300}{mr} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-master-2][10.42.3.74:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:966) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.9.0.jar:7.9.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.9.0.jar:7.9.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.42.3.74/10.42.3.74:9300
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
... 7 more
[2021-05-04T14:38:55,554][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] master node changed {previous [], current [{elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr}]}, term: 31, version: 181, reason: ApplyCommitRequest{term=31, version=181, sourceNode={elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr}}
[2021-05-04T14:38:55,629][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] removed {{elasticsearch-master-2}{PTtLrgqkSzSeWh0OfJMpTg}{Ung6LwKlTNSRY4lgnndmTA}{10.42.3.74}{10.42.3.74:9300}{mr}}, term: 31, version: 182, reason: ApplyCommitRequest{term=31, version=182, sourceNode={elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr}}
[2021-05-04T14:38:56,101][INFO ][o.e.c.c.Coordinator ] [elasticsearch-master-1] master node [{elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [elasticsearch-master-0][10.42.3.77:9300][disconnected] disconnected
[2021-05-04T14:38:56,104][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] master node changed {previous [{elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr}], current []}, term: 31, version: 182, reason: becoming candidate: onLeaderFailure
[2021-05-04T14:38:56,123][WARN ][o.e.c.NodeConnectionsService] [elasticsearch-master-1] failed to connect to {elasticsearch-master-0}{jZdZNvDxQU64pgrwDEW4xA}{aJmnS_kHQXqufQphOtttXA}{10.42.3.77}{10.42.3.77:9300}{mr} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-master-0][10.42.3.77:9300] connect_exception