# Is it normal for cluster to turn "Yellow" daily briefly during heavy indexing to existing index? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 10, 2020, 3:02am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098 "2020-09-10T03:02:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![vorapoap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vorapoap/32/66582_2.png) [@vorapoap](https://discuss.elastic.co/u/vorapoap)\ **Post date:** [September 10, 2020, 3:02am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098/1 "2020-09-10T03:02:17Z")
Cluster configuration: 3 Dedicated Master Node 15 Data nodes 0-all Replica (meaning the replica on every data node) The index is pre created. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 10, 2020, 3:25am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098/2 "2020-09-10T03:25:45Z")
What do your master node logs show at that time? ---
**Author:** ![vorapoap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vorapoap/32/66582_2.png) [@vorapoap](https://discuss.elastic.co/u/vorapoap)\ **Post date:** [September 10, 2020, 9:16am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098/3 "2020-09-10T09:16:03Z")
What could cause brief node connection refused and disconnection?... I see several of this in the log almost daily. ``` [2020-09-06T16:22:22,735][WARN][o.e.c.NodeConnectionsService] [63fa72c3cf1617351c1153d84cc0b32f] failed to connect to node {bfc1233fa07b84e29aab78adbda03dac}{hwiWhedWQjqx4NAH32ssbw}{AMzZJ5OmQYOQVYsHY2wjbg} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [bfc1233fa07b84e29aab78adbda03dac][__IP__] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1.jar:7.1.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1.jar:7.1.1] at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] ... 6 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] ... 6 more ``` I see few of this in the log... seems like once a week.. ``` [WARN][o.e.a.b.TransportShardBulkAction] [01a52c4d0d89aedc987c4b5434f5c241] [[....][6]] failed to perform __PATH__ [s] on replica [.....][6], node[hwiWhedWQjqx4NAH32ssbw], [R], s[STARTED], a[id=6I2X5rwnTziAofQRRhASbg] org.elasticsearch.transport.NodeDisconnectedException: [bfc1233fa07b84e29aab78adbda03dac][__IP__][__PATH__ [s][r]] disconnected| |@timestamp|1599679820410| ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 10, 2020, 9:59am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098/4 "2020-09-10T09:59:19Z")
> [@vorapoap](#): > > ```auto > Caused by: java.net.ConnectException: Connection refused > > ``` That could be from a firewall with a connection lifetime limit. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 8, 2020, 9:59am UTC](https://discuss.elastic.co/t/is-it-normal-for-cluster-to-turn-yellow-daily-briefly-during-heavy-indexing-to-existing-index/248098/5 "2020-10-08T09:59:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.