# ElasticSearch cluster unreachable every night between 00.00 and 02.00

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249>\
**Category:** Elasticsearch\
**Created:** [June 16, 2020, 8:08am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249 "2020-06-16T08:08:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zeroc0pter](https://avatars.discourse-cdn.com/v4/letter/z/34f0e0/32.png) [@Zeroc0pter](https://discuss.elastic.co/u/Zeroc0pter)\
**Post date:** [June 16, 2020, 8:08am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/1 "2020-06-16T08:08:06Z")

</div>

Hi fellows,

I have a question regarding the 'possible' scheduled jobs of ElasticSeach. We monitor the application with cURL and run it every 30 seconds to inspect the HTTPS response. Somehow, we receive an unavailable error response every night between 00.00 - 02.00.

Is there a scheduled task that does something really intense that blocks incoming traffic?

I hope someone could help us out.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 16, 2020, 8:12am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/2 "2020-06-16T08:12:26Z")

</div>

Welcome to our community! 😃

What do your Elasticsearch logs show at that time?

---

<div class="post-metadata">

**Author:** ![Zeroc0pter](https://avatars.discourse-cdn.com/v4/letter/z/34f0e0/32.png) [@Zeroc0pter](https://discuss.elastic.co/u/Zeroc0pter)\
**Post date:** [June 16, 2020, 8:13am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/3 "2020-06-16T08:13:37Z")

</div>

Nothing strange ☹

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 16, 2020, 8:14am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/4 "2020-06-16T08:14:35Z")

</div>

There's nothing in Elasticsearch that would cause this, and if the logs are not showing excessive GC or node disconnections, then it's external to the cluster.

---

<div class="post-metadata">

**Author:** ![Zeroc0pter](https://avatars.discourse-cdn.com/v4/letter/z/34f0e0/32.png) [@Zeroc0pter](https://discuss.elastic.co/u/Zeroc0pter)\
**Post date:** [June 16, 2020, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/5 "2020-06-16T08:15:36Z")

</div>

Thanks. let me download the log files and inspect them again. I use the following files:

- gc.log
- es-cluster.log

edit:  
I have found the following lines:

````auto
[2020-06-16T02:00:01,373][INFO][o.e.c.r.a.AllocationService] [es-server01-prod.aws.company.domain] updating number_of_replicas to [1] for indices [.monitoring-es-6-2020.06.16]
[2020-06-16T02:00:03,083][INFO][o.e.c.r.a.AllocationService] [es-server01-prod.aws.company.domain] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2020.06.16][0]] ...]).
[2020-06-16T02:00:07,539][INFO][o.e.c.m.MetaDataMappingService] [es-server01-prod.aws.company.domain] [filebeat-2020.06.16/DeG8CocARomBTmxXp2LpXQ] update_mapping [doc]
[2020-06-16T02:00:08,451][INFO][o.e.c.m.MetaDataCreateIndexService] [es-server01-prod.aws.company.domain] [.monitoring-kibana-6-2020.06.16] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0], mappings [doc]
[2020-06-16T02:00:08,452][INFO][o.e.c.r.a.AllocationService] [es-server01-prod.aws.company.domain] updating number_of_replicas to [1] for indices [.monitoring-kibana-6-2020.06.16]
[2020-06-16T02:00:10,216][INFO][o.e.c.r.a.AllocationService] [es-server01-prod.aws.company.domain] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-kibana-6-2020.06.16][0]] ...]).```
````

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2020, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-unreachable-every-night-between-00-00-and-02-00/237249/6 "2020-07-14T08:15:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.