# High load at midnight caused by ES docker containers while no usage occurs

**URL:** <https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [July 26, 2022, 3:11pm UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663 "2022-07-26T15:11:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tamaloa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tamaloa/32/108859_2.png) [@tamaloa](https://discuss.elastic.co/u/tamaloa)\
**Post date:** [July 26, 2022, 3:11pm UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663/1 "2022-07-26T15:11:50Z")

</div>

We are running a somewhat untypical setup (not your usual production elasticsearch usage) of around 40 very small (single node) elasticsearch docker instances (version 7.17.3) per server.  
Most instances contain very little data even the biggest ones are only a few hundred MB in disk size.

Every night at midnight (local time) we experience extremely high load caused by the ES docker containers. For about 20 seconds the CPUs are saturated and RAM is maxed out, then for about 10 seconds high disk write occurs.  
The load has been high enough to occasionally freeze our (reasonably sized) server for a few minutes. The only workaround we have found is to spread out our applications to more servers but of course otherwise the servers are moreless idle.

Are there any daily cleanup jobs in elasticsearch (similar to postgres auto vacuum etc.) which are triggered by default at midnight?  
If yes - is it possible to configure a different default start time (which would solve our issues)?  
If no - how to find the culprit? Is there some kind of monitoring log which can be activated to catch anything causing such high load inside the ES containers?

A example monitoring graph from one of our severs.

 ![Screenshot from 2022-07-26 17-04-00](https://us1.discourse-cdn.com/elastic/original/3X/a/a/aa278544390aad31d21c9f692f55292a411f95db.png)

This is the exact docker configuration we are using:

```auto
  elasticsearch:
    # https://www.docker.elastic.co/r/elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.3
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - http.cors.allow-origin=http://localhost:1358,http://127.0.0.1:1358
      - http.cors.enabled=true
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - http.cors.allow-credentials=true
      - cluster.routing.allocation.disk.threshold_enabled=false # Disable watermark check (safeguard which sets write only indexes on low disk space)
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/elastic01:/usr/share/elasticsearch/data

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 27, 2022, 1:41am UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663/2 "2022-07-27T01:41:15Z")

</div>

Welcome to our community! 😃

> [@tamaloa](#):
>
> Are there any daily cleanup jobs in elasticsearch (similar to postgres auto vacuum etc.) which are triggered by default at midnight?

Not generically.

> [@tamaloa](#):
>
> ```auto
> - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
> 
> ```

This is super small and it doesn't make sense to have 40 nodes of this side, what's the reason for it?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 27, 2022, 2:11am UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663/3 "2022-07-27T02:11:41Z")

</div>

If you are doing ILM with Force Merge etc... it could be running each day around the same time... that can consume resources...

Ahh my good friend Postgres's vacuuming.... 🙂

---

<div class="post-metadata">

**Author:** ![tamaloa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tamaloa/32/108859_2.png) [@tamaloa](https://discuss.elastic.co/u/tamaloa)\
**Post date:** [July 27, 2022, 10:48am UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663/4 "2022-07-27T10:48:21Z")

</div>

Thanks for your answers. It was a long shot, but good to know it should not be something built into elasticsearch by default (I don't even know what ILM is so I assume we arn't using it).

Regarding the high number of nodes and memory limitations: We are basically running one docker container (and elasticsearch service container) per customer due to different security and privacy concerns (and because multi-tenancy is not yet implemented). As this is a customer onboarding/trial environment we did not have any problems with the memory settings (before elastic containers occupied quiete a lot of memory).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 24, 2022, 10:48am UTC](https://discuss.elastic.co/t/high-load-at-midnight-caused-by-es-docker-containers-while-no-usage-occurs/310663/5 "2022-08-24T10:48:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.