# Publication of cluster state fails - followers check retry count exceeded **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [April 17, 2023, 7:22am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097 "2023-04-17T07:22:01Z")\ **Posts on this page:** 11\ **Page:** 3
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 18, 2023, 12:06pm UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/41 "2023-05-18T12:06:31Z")
A `disconnected` requires that the shutdown happens somewhat gracefully, but when a spot instance dies it may just abruptly drop off the network without an explicit disconnect. Remember that the fundamental problem here seems to be that nodes are inexplicably shutting down, apparently without logging anything. That's usually the OOM killer, but spot instances would explain it too. This is the first mention of spot instances in this thread. ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [May 18, 2023, 12:19pm UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/42 "2023-05-18T12:19:01Z")
ok @DavidTurner - we'll give it a try. We are going to switch the `data-wrk` nodes to on-demend for the upcoming days. I will update if that makes any change. Thank you ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [May 22, 2023, 9:36am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/43 "2023-05-22T09:36:08Z")
@DavidTurner it was it - spot instances ☹ It was tricky because in another cluster we used them and it was super stable (they were living for days). After switching to on-demand all of our `data-wrk` are stable (4 days now) 🙂 Thanks a LOT! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 22, 2023, 9:48am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/44 "2023-05-22T09:48:33Z")
Makes sense, thanks for confirming 🙂 I think you're probably also not configuring the [TCP retransmission timeout as the manual recommends](https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config-tcpretries.html). If you do that then you would get a `disconnected` message more reliably when a node vanishes, as well as the other benefits described in those docs. But yes, spot instances don't work very well with stateful services like Elasticsearch. ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [June 5, 2023, 3:23pm UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/45 "2023-06-05T15:23:58Z")
We've experienced the same errors in our cluster and data nodes re-joining the cluster. In our case the root cause of the issue was swap accidentally turned on. Our root disks are slow AWS EBS disks, so looks like nodes were "freezing" during swap operations and fault detection kicked them from the cluster. ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [June 18, 2023, 9:49am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/46 "2023-06-18T09:49:01Z")
thanks @kkn87 - how did you find it? (I want to make sure mine configured correctly) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 18, 2023, 10:26am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/47 "2023-06-18T10:26:07Z")
Heavy/slow swapping wouldn't lead to the same symptoms as described in this thread. It may cause nodes to leave the cluster but they would normally rejoin _without_ restarting, having logged messages about slow GC or blocked threads. Definitely check that swap is disabled, but I don't think it's related. ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [June 19, 2023, 5:40am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/48 "2023-06-19T05:40:39Z")
thanks @DavidTurner - what's the best way to ensure that? 🙂 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2023, 7:38am UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/49 "2023-06-19T07:38:13Z")
[How can I check if swap is active from the command line? - Unix & Linux Stack Exchange](https://unix.stackexchange.com/questions/23072/how-can-i-check-if-swap-is-active-from-the-command-line#23074) gives a good selection of ways to do that ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [June 20, 2023, 4:31pm UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/50 "2023-06-20T16:31:34Z")
I've discovered high swapping during node outages using atop tool ([GitHub - Atoptool/atop: System and process monitor for Linux](https://github.com/Atoptool/atop)), but any monitoring tool can help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2023, 4:31pm UTC](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097/51 "2023-07-18T16:31:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/publication-of-cluster-state-fails-followers-check-retry-count-exceeded/330097.md?page=2)