# Nodes constantly losing connection

**URL:** <https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331>\
**Category:** Elasticsearch\
**Created:** [January 27, 2021, 7:42am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331 "2021-01-27T07:42:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [January 27, 2021, 7:42am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331/1 "2021-01-27T07:42:31Z")

</div>

I have a 2-node cluster(archive and main) and today they start losing connection.  
These are the two logs that keep appearing:  
`[2021-01-27T10:26:55,816][WARN][o.e.g.PersistedClusterStateService] [XXXarchive.XXX.local] writing cluster state took [69043ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [124] indices and skipped [127] unchanged indices [2021-01-27T10:28:11,586][WARN][o.e.c.c.C.CoordinatorPublication] [XXX-main.XXX.local] after [30s] publication of cluster state version [95968] is still waiting for {XXXarchive.XXX.local}{cPA9OW7KQhKGQ-_xtXGnhg}{y-JIm7z-QWO8NhP2fdwTBQ}{XXX-archive.XXX.local}{X.X.X.103:9300}{dilmrt}{ml.machine_memory=8201244672, ml.max_open_jobs=20, xpack.installed=true, box_type=warm, transform.node=true} [SENT_PUBLISH_REQUEST] `  
I have around 140 indices and 160 shards.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 27, 2021, 7:46am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331/2 "2021-01-27T07:46:47Z")

</div>

> [@borna\_talebi](#):
>
> writing cluster state took [69043ms] which is above the warn threshold of [10s]

It looks like your cluster is overloaded and/or has far too slow storage. What is the specification of your cluster? What type of storage are you using?

---

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [January 27, 2021, 9:05am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331/3 "2021-01-27T09:05:02Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> What type of storage are you using?

I'm using a SAN HDD on main node and an NFS disk on the second(archive) node. I know they're pretty slow but I've never had this problem in the past 3 months.

Here are the putput of `iostat` on both nodes:  
main:

 ![Screenshot from 2021-01-27 12-31-09](https://us1.discourse-cdn.com/elastic/original/3X/0/7/073b027539cf28c3d25baeb81bf0bbb04f377dc0.png)  
archive:  
 ![Screenshot from 2021-01-27 12-34-24](https://us1.discourse-cdn.com/elastic/original/3X/7/0/708627c62e41cf6ec741a641b1313aacde1dd858.png)

Could the high **iowait** percentage be a problem?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 27, 2021, 10:30am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331/4 "2021-01-27T10:30:37Z")

</div>

> [@borna\_talebi](#):
>
> I'm using a SAN HDD on main node and an NFS disk on the second(archive) node. I know they're pretty slow but I've never had this problem in the past 3 months.

That does not mean that it is not a problem now.

> [@borna\_talebi](#):
>
> Could the high **iowait** percentage be a problem?

Yes. It looks like this node is struggling, at least periodically.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 24, 2021, 10:30am UTC](https://discuss.elastic.co/t/nodes-constantly-losing-connection/262331/5 "2021-02-24T10:30:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.