I have updated the cluster with the basic license as suggested and the License related logs messages are not appearing anymore in the logs.
Below are the various details with IP's.
Load Balancer -
lb-1 172.19.122.4
lb-2 172.19.62.65
4 Node ES cluster -
node-1 172.19.68.62 --> master
node-2 172.19.68.215
node-3 172.19.2.18
node-4 172.19.82.19
I tried doing curl 172.19.2.18:9200/_cat/nodes bypassing the LB, but i didn't get any response even though i have waited for ~15 mins!!! I have tried with master node as well with same result.
When i changed the Log level to Debug, one log messages which i see consistently across the cluster where this issue exists is as below:
caught exception while handling client http traffic, closing connection [id: 0x47397fe6, L:/172.19.2.18:9200 - R:/172.19.122.4:52492]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.13.Final.jar:4.1.13.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100) ~[netty-buffer-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
This appears even if i do not hit the endpoints where we are seeing issue(viz., _nodes or _nodes/stats)
Let me know your thoughts.
Thank You.