# Every Night- Elasticsearch. (code=killed, signal=KILL)

**URL:** <https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397>\
**Category:** Elasticsearch\
**Created:** [January 14, 2022, 10:26am UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397 "2022-01-14T10:26:29Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![beohna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beohna/32/100330_2.png) [@beohna](https://discuss.elastic.co/u/beohna)\
**Post date:** [January 14, 2022, 10:26am UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/1 "2022-01-14T10:26:29Z")

</div>

Hey,

on my Rasperry Pi with Nextcloud and full text search (powered by Elasticsearch) every night  
Elasticsearch is killed (code=killed, signal=KILL).

```auto
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2022-01-13 01:27:21 CET; 1 day 8h ago
       Docs: https://www.elastic.co
    Process: 670366 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
   Main PID: 670366 (code=killed, signal=KILL)
        CPU: 20min 11.324s

Jan 12 07:49:17 Server systemd[1]: Starting Elasticsearch...
Jan 12 07:50:13 Server systemd[1]: Started Elasticsearch.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Unit process 670588 (controller) remains running after unit stopped.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Consumed 20min 11.308s CPU time.

```

Here the gc.log.23

```auto
[2022-01-14T10:20:17.766+0000][2176][gc] Using G1
[2022-01-14T10:20:17.808+0000][2176][gc,init] Version: 17.0.1+12 (release)
[2022-01-14T10:20:17.808+0000][2176][gc,init] CPUs: 4 total, 4 available
[2022-01-14T10:20:17.808+0000][2176][gc,init] Memory: 3844M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Large Page Support: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] NUMA Support: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] Compressed Oops: Enabled (32-bit)
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Region Size: 1M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Min Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Initial Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Max Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Pre-touch: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] Parallel Workers: 4
[2022-01-14T10:20:17.808+0000][2176][gc,init] Concurrent Workers: 1
[2022-01-14T10:20:17.808+0000][2176][gc,init] Concurrent Refinement Workers: 4
[2022-01-14T10:20:17.808+0000][2176][gc,init] Periodic GC: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] CDS archive(s) not mapped
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] Compressed class space mapped at: 0x0000000100000000-0x0000000140000000, reserved size: 1073741824
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x140000000
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] Heap
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] garbage-first heap total 1968128K, used 1538K [0x0000000087e00000, 0x0000000100000000)
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] region size 1024K, 2 young (2048K), 0 survivors (0K)
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] Metaspace used 3427K, committed 3520K, reserved 1056768K
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] class space used 269K, committed 320K, reserved 1048576K

```

Any idea?  
Greetings  
beohna

* * *

My environment:  
Rasperry Pi - model : RPi 4 Model B (aarch64) 4GB RAM  
DietPi v7.9.3  
Nextcloud v23

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 14, 2022, 1:15pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/2 "2022-01-14T13:15:09Z")

</div>

You need to share Elasticsearch logs and also system logs from the time Elasticsearch stopped.

---

<div class="post-metadata">

**Author:** ![beohna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beohna/32/100330_2.png) [@beohna](https://discuss.elastic.co/u/beohna)\
**Post date:** [January 15, 2022, 5:36pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/3 "2022-01-15T17:36:14Z")

</div>

Here the system log from the time Elasticsearch stopped.

journalctl --since "2022-01-15 01:30" --until "2022-01-15 02:00"

```auto
Jan 15 01:52:27 Server kernel: rpimonitord invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 15 01:52:27 Server kernel: CPU: 2 PID: 904 Comm: rpimonitord Not tainted 5.10.63-v8+ #1488
Jan 15 01:52:27 Server kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jan 15 01:52:27 Server kernel: Call trace:
Jan 15 01:52:27 Server kernel: dump_backtrace+0x0/0x1b8
Jan 15 01:52:27 Server kernel: show_stack+0x20/0x30
Jan 15 01:52:27 Server kernel: dump_stack+0xf0/0x158
Jan 15 01:52:27 Server kernel: dump_header+0x4c/0x204
Jan 15 01:52:27 Server kernel: oom_kill_process+0x1f8/0x200
Jan 15 01:52:27 Server kernel: out_of_memory+0xf0/0x330
Jan 15 01:52:27 Server kernel: __alloc_pages_slowpath.constprop.133+0x820/0xb80
Jan 15 01:52:27 Server kernel: __alloc_pages_nodemask+0x2bc/0x330
Jan 15 01:52:27 Server kernel: pagecache_get_page+0x144/0x2e0
Jan 15 01:52:27 Server kernel: filemap_fault+0x520/0x9e0
Jan 15 01:52:27 Server kernel: ext4_filemap_fault+0x3c/0x818
Jan 15 01:52:27 Server kernel: __do_fault+0x44/0x110
Jan 15 01:52:27 Server kernel: handle_mm_fault+0x6a8/0xcb8
Jan 15 01:52:27 Server kernel: do_page_fault+0x148/0x3e8
Jan 15 01:52:27 Server kernel: do_translation_fault+0x58/0x70
Jan 15 01:52:27 Server kernel: do_mem_abort+0x48/0xa8
Jan 15 01:52:27 Server kernel: el0_ia+0x68/0xc8
Jan 15 01:52:27 Server kernel: el0_sync_handler+0x78/0xb8
Jan 15 01:52:27 Server kernel: el0_sync+0x180/0x1c0
Jan 15 01:52:27 Server kernel: Mem-Info:
Jan 15 01:52:27 Server kernel: active_anon:263544 inactive_anon:694226 isolated_anon:0
                                active_file:231 inactive_file:0 isolated_file:0
                                unevictable:387 dirty:0 writeback:0
                                slab_reclaimable:5894 slab_unreclaimable:5440
                                mapped:7320 shmem:275152 pagetables:2615 bounce:0
                                free:6987 free_pcp:245 free_cma:468
Jan 15 01:52:27 Server kernel: Node 0 active_anon:1054176kB inactive_anon:2776904kB active_file:924kB inactive_file:0kB unevictable:1548kB isolated(anon):0kB isolated(file):0kB mapped:29280kB dirty:0kB writeback:0kB shmem:1100608kB writeback_tmp:0kB kernel_stack:3664kB all_unreclaimable? no
Jan 15 01:52:27 Server kernel: DMA free:15576kB min:3900kB low:4872kB high:5844kB reserved_highatomic:0KB active_anon:600340kB inactive_anon:306160kB active_file:0kB inactive_file:220kB unevictable:0kB writepending:0kB present:1021952kB managed:937872kB mlocked:0kB pagetables:1768kB bounce:0kB free_pcp:960kB local_pcp:280kB free_cma:1872kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 2929 2929 2929
Jan 15 01:52:27 Server kernel: DMA32 free:12372kB min:12480kB low:15600kB high:18720kB reserved_highatomic:0KB active_anon:453836kB inactive_anon:2470744kB active_file:0kB inactive_file:284kB unevictable:1548kB writepending:0kB present:3080192kB managed:2999332kB mlocked:1548kB pagetables:8692kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 0 0 0
Jan 15 01:52:27 Server kernel: DMA: 592*4kB (UMEC) 327*8kB (UMEC) 129*16kB (UEC) 102*32kB (UEC) 44*64kB (UEC) 10*128kB (UEC) 0*256kB 3*512kB (C) 0*1024kB 0*2048kB 0*4096kB = 15944kB
Jan 15 01:52:27 Server kernel: DMA32: 409*4kB (UME) 548*8kB (UME) 208*16kB (UME) 43*32kB (UM) 30*64kB (UME) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12900kB
Jan 15 01:52:27 Server kernel: 275866 total pagecache pages
Jan 15 01:52:27 Server kernel: 0 pages in swap cache
Jan 15 01:52:27 Server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 15 01:52:27 Server kernel: Free swap = 0kB
Jan 15 01:52:27 Server kernel: Total swap = 0kB
Jan 15 01:52:27 Server kernel: 1025536 pages RAM
Jan 15 01:52:27 Server kernel: 0 pages HighMem/MovableOnly
Jan 15 01:52:27 Server kernel: 41235 pages reserved
Jan 15 01:52:27 Server kernel: 16384 pages cma reserved
Jan 15 01:52:27 Server kernel: Tasks state (memory values in pages):
Jan 15 01:52:27 Server kernel: [pid] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 15 01:52:27 Server kernel: [137] 0 137 6115 597 73728 0 -250 systemd-journal
Jan 15 01:52:27 Server kernel: [185] 0 185 5016 282 49152 0 -1000 systemd-udevd
Jan 15 01:52:27 Server kernel: [306] 0 306 884 358 32768 0 0 vmtouch
Jan 15 01:52:27 Server kernel: [313] 0 313 749 26 36864 0 0 rngd
Jan 15 01:52:27 Server kernel: [334] 0 334 1074 48 36864 0 0 dropbear
Jan 15 01:52:27 Server kernel: [405] 0 405 24837 325 61440 0 0 dhclient
Jan 15 01:52:27 Server kernel: [471] 101 471 22661 850 73728 0 0 systemd-timesyn
Jan 15 01:52:27 Server kernel: [492] 0 492 3107 966 45056 0 0 dietpi-dashboar
Jan 15 01:52:27 Server kernel: [502] 0 502 1017 26 36864 0 0 agetty
Jan 15 01:52:27 Server kernel: [503] 0 503 1017 28 40960 0 0 agetty
Jan 15 01:52:27 Server kernel: [673] 0 673 18129 596 118784 0 0 nmbd
Jan 15 01:52:27 Server kernel: [717] 0 717 22096 1168 155648 0 0 smbd
Jan 15 01:52:27 Server kernel: [719] 0 719 21615 735 147456 0 0 smbd-notifyd
Jan 15 01:52:27 Server kernel: [720] 0 720 21617 737 143360 0 0 cleanupd
Jan 15 01:52:27 Server kernel: [723] 105 723 15628 592 102400 0 0 redis-server
Jan 15 01:52:27 Server kernel: [781] 104 781 454773 49713 626688 0 0 mariadbd
Jan 15 01:52:27 Server kernel: [823] 0 823 88563 2119 176128 0 0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [880] 33 880 2480 659 49152 0 0 lighttpd
Jan 15 01:52:27 Server kernel: [902] 0 902 18217 4060 126976 0 0 rpimonitord
Jan 15 01:52:27 Server kernel: [903] 1000 903 18349 4198 126976 0 0 rpimonitord
Jan 15 01:52:27 Server kernel: [904] 0 904 18264 4069 126976 0 0 rpimonitord
Jan 15 01:52:27 Server kernel: [914] 0 914 1413 64 40960 0 0 cron
Jan 15 01:52:27 Server kernel: [2020] 106 2020 1471905 600355 5423104 0 0 java
Jan 15 01:52:27 Server kernel: [2221] 106 2221 25951 142 77824 0 0 controller
Jan 15 01:52:27 Server kernel: [5386] 0 5386 1140 96 36864 0 0 dropbear
Jan 15 01:52:27 Server kernel: [21156] 0 21156 1949 625 49152 0 0 bash
Jan 15 01:52:27 Server kernel: [21187] 0 21187 3552 422 61440 0 0 mc
Jan 15 01:52:27 Server kernel: [21189] 0 21189 1948 625 40960 0 0 bash
Jan 15 01:52:27 Server kernel: [21501] 0 21501 2014 694 49152 0 0 bash
Jan 15 01:52:27 Server kernel: [24723] 33 24723 89797 8935 233472 0 0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [24774] 33 24774 114377 8663 249856 0 0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [24838] 33 24838 89776 8635 225280 0 0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [24854] 33 24854 89285 8123 221184 0 0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [57209] 0 57209 2228 116 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57210] 33 57210 517 17 36864 0 0 sh
Jan 15 01:52:27 Server kernel: [57211] 33 57211 63301 9284 225280 0 0 php
Jan 15 01:52:27 Server kernel: [57243] 0 57243 2136 105 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57251] 0 57251 2136 105 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57252] 0 57252 2136 103 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57256] 0 57256 2136 103 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57258] 0 57258 2136 103 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57264] 0 57264 2136 103 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57271] 0 57271 2136 103 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57275] 0 57275 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57278] 0 57278 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57280] 0 57280 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57285] 0 57285 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57288] 0 57288 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57290] 0 57290 2136 90 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57295] 0 57295 2136 89 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57296] 0 57296 2136 89 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57301] 0 57301 2027 79 49152 0 0 cron
Jan 15 01:52:27 Server kernel: [57304] 0 57304 22136 760 139264 0 0 smbd
Jan 15 01:52:27 Server kernel: [57305] 0 57305 1637 72 45056 0 0 cron
Jan 15 01:52:27 Server kernel: [57309] 0 57309 1534 68 45056 0 0 cron
Jan 15 01:52:27 Server kernel: [57310] 0 57310 1534 68 45056 0 0 cron
Jan 15 01:52:27 Server kernel: [57314] 0 57314 18129 596 114688 0 0 nmbd
Jan 15 01:52:27 Server kernel: [57315] 0 57315 1413 64 40960 0 0 cron
Jan 15 01:52:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=2020,uid=106
Jan 15 01:52:27 Server kernel: Out of memory: Killed process 2020 (java) total-vm:5887620kB, anon-rss:2401228kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5296kB oom_score_adj:0
Jan 15 01:52:27 Server kernel: oom_reaper: reaped process 2020 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Unit process 2221 (controller) remains running after unit stopped.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Consumed 22min 47.135s CPU time.
Jan 15 01:52:23 Server systemd[1]: Starting Clean php session files...
Jan 15 01:52:23 Server CRON[57252]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57295]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57271]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57310]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57305]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57288]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57315]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57258]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57296]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57301]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57264]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57319]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57280]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57275]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57243]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57256]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57290]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57285]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57325]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57328]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57327]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57326]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Jan 15 01:52:23 Server CRON[57335]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57324]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57336]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57330]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57333]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57337]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57329]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57332]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57334]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57331]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57338]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57339]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57340]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57341]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57342]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 15 01:52:23 Server CRON[57344]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57343]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:24 Server systemd[1]: Stopping Network Time Synchronization...
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Succeeded.
Jan 15 01:52:24 Server systemd[1]: Stopped Network Time Synchronization.
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Consumed 48.219s CPU time.
Jan 15 01:52:24 Server CRON[57288]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:25 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 01:52:25 Server systemd[1]: Finished Clean php session files.
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:25 Server runuser[57574]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:26 Server runuser[57574]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:26 Server CRON[57295]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:29 Server CRON[57252]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57285]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57310]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57319]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57271]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57296]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57275]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57301]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57305]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57315]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57256]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57290]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57258]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57243]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57280]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:55:02 Server CRON[57710]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:55:02 Server CRON[57711]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:55:03 Server CRON[57710]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:56:25 Server CRON[57209]: pam_unix(cron:session): session closed for user www-data

```

---

<div class="post-metadata">

**Author:** ![beohna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beohna/32/100330_2.png) [@beohna](https://discuss.elastic.co/u/beohna)\
**Post date:** [January 15, 2022, 5:38pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/4 "2022-01-15T17:38:24Z")

</div>

service Elasticsearch status

```auto
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Sat 2022-01-15 01:52:23 CET; 14h ago
       Docs: https://www.elastic.co
    Process: 2020 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
   Main PID: 2020 (code=killed, signal=KILL)
        CPU: 22min 47.145s

Jan 14 11:20:14 Server systemd[1]: Starting Elasticsearch...
Jan 14 11:21:14 Server systemd[1]: Started Elasticsearch.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Unit process 2221 (controller) remains running after unit stopped.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Consumed 22min 47.135s CPU time.

```

And sorry the Elasticsearch.log is empty.

Greetings  
beohna

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 15, 2022, 6:18pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/5 "2022-01-15T18:18:57Z")

</div>

The process is being killed by Out of Memory reasons:

```auto
Jan 15 01:52:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=2020,uid=106
Jan 15 01:52:27 Server kernel: Out of memory: Killed process 2020 (java) total-vm:5887620kB, anon-rss:2401228kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5296kB oom_score_adj:0
Jan 15 01:52:27 Server kernel: oom_reaper: reaped process 2020 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB

```

These OOM issues can happen for many reasons, one is having too much data for a small node and make expensive queries.

Normally the solution would be to increase the memory of the server and the service, but since you are running with limited resources on a Raspberry Pi, you can't do that.

What you can try to do is to check the number of indices and shards you have in the node and see if you can reduce it to make your single-node cluster more stable, but I do not know how Nextcloud uses Elasticsearch.

You can check the number of shards using the following curl request from your node:

```auto
curl -X GET "localhost:9200/_cat/shards?pretty"

```

Can you run that and share the result?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 16, 2022, 9:03am UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/6 "2022-01-16T09:03:26Z")

</div>

> [@leandrojmp](#):
>
> These OOM issues can happen for many reasons, one is having too much data for a small node and make expensive queries.

Note that this is the OS `oom-killer`, not a JVM `OutOfMemoryException` so the problem is not that Elasticsearch ran out of heap (too much data etc) it's that Elasticsearch _plus everything else on the same machine_ consumed too much memory. Either use a smaller heap or reduce the needs of the other things that this machine is running.

---

<div class="post-metadata">

**Author:** ![beohna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beohna/32/100330_2.png) [@beohna](https://discuss.elastic.co/u/beohna)\
**Post date:** [January 16, 2022, 12:26pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/7 "2022-01-16T12:26:13Z")

</div>

curl -X GET "localhost:9200/\_cat/shards?pretty":

```auto
.ds-ilm-history-5-2022.01.12-000002 0 p STARTED 127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.28-000002 0 p STARTED 127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.28-000002 0 r UNASSIGNED
.geoip_databases 0 p STARTED 42 52.2mb 127.0.0.1 Server
.ds-ilm-history-5-2021.12.13-000001 0 p STARTED 127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2022.01.11-000003 0 p STARTED 127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2022.01.11-000003 0 r UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 p STARTED 127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 r UNASSIGNED
cloud.XXXXXX.com 0 p STARTED 44823 163.6mb 127.0.0.1 Server
cloud.XXXXXX.com 0 r UNASSIGNED

```

This is the result after a restart of the Elasticsearch service.

I also uninstall the RPi-Monitor, I think this can be the reason of the memory problem.

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![beohna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beohna/32/100330_2.png) [@beohna](https://discuss.elastic.co/u/beohna)\
**Post date:** [January 17, 2022, 8:51pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/8 "2022-01-17T20:51:38Z")

</div>

next night...

journalctl --since "2022-01-17 00:30" --until "2022-01-17 02:00"

```auto
Jan 17 00:50:41 Server kernel: php invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 17 00:50:42 Server kernel: CPU: 2 PID: 26275 Comm: php Not tainted 5.10.63-v8+ #1488
Jan 17 00:50:42 Server kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jan 17 00:50:42 Server kernel: Call trace:
Jan 17 00:50:42 Server kernel: dump_backtrace+0x0/0x1b8
Jan 17 00:50:42 Server kernel: show_stack+0x20/0x30
Jan 17 00:50:42 Server kernel: dump_stack+0xf0/0x158
Jan 17 00:50:42 Server kernel: dump_header+0x4c/0x204
Jan 17 00:50:42 Server kernel: oom_kill_process+0x1f8/0x200
Jan 17 00:50:42 Server kernel: out_of_memory+0xf0/0x330
Jan 17 00:50:42 Server kernel: __alloc_pages_slowpath.constprop.133+0x820/0xb80
Jan 17 00:50:42 Server kernel: __alloc_pages_nodemask+0x2bc/0x330
Jan 17 00:50:42 Server kernel: pagecache_get_page+0x144/0x2e0
Jan 17 00:50:42 Server kernel: filemap_fault+0x520/0x9e0
Jan 17 00:50:42 Server kernel: ext4_filemap_fault+0x3c/0x818
Jan 17 00:50:42 Server kernel: __do_fault+0x44/0x110
Jan 17 00:50:42 Server kernel: handle_mm_fault+0x6a8/0xcb8
Jan 17 00:50:42 Server kernel: do_page_fault+0x148/0x3e8
Jan 17 00:50:42 Server kernel: do_translation_fault+0x58/0x70
Jan 17 00:50:42 Server kernel: do_mem_abort+0x48/0xa8
Jan 17 00:50:42 Server kernel: el0_da+0x40/0x58
Jan 17 00:50:42 Server kernel: el0_sync_handler+0x68/0xb8
Jan 17 00:50:42 Server kernel: el0_sync+0x180/0x1c0
Jan 17 00:50:42 Server kernel: Mem-Info:
Jan 17 00:50:42 Server kernel: active_anon:277490 inactive_anon:680911 isolated_anon:0
                                active_file:170 inactive_file:0 isolated_file:0
                                unevictable:387 dirty:0 writeback:0
                                slab_reclaimable:5708 slab_unreclaimable:5063
                                mapped:7337 shmem:288966 pagetables:2359 bounce:0
                                free:7405 free_pcp:124 free_cma:386
Jan 17 00:50:42 Server kernel: Node 0 active_anon:1109960kB inactive_anon:2723644kB active_file:680kB inactive_file:0kB unevictable:1548kB isolated(anon):0kB isolated(file):0kB mapped:29348kB dirty:0kB writeback:0kB shmem:1155864kB writeback_tmp:0kB kernel_stack:3376kB all_unreclaimable? no
Jan 17 00:50:42 Server kernel: DMA free:17144kB min:3900kB low:4872kB high:5844kB reserved_highatomic:0KB active_anon:668892kB inactive_anon:240396kB active_file:0kB inactive_file:304kB unevictable:0kB writepending:0kB present:1021952kB managed:937872kB mlocked:0kB pagetables:1092kB bounce:0kB free_pcp:496kB local_pcp:248kB free_cma:1544kB
Jan 17 00:50:42 Server kernel: lowmem_reserve[]: 0 2929 2929 2929
Jan 17 00:50:42 Server kernel: DMA32 free:12476kB min:12480kB low:15600kB high:18720kB reserved_highatomic:0KB active_anon:441068kB inactive_anon:2483248kB active_file:0kB inactive_file:128kB unevictable:1548kB writepending:0kB present:3080192kB managed:2999332kB mlocked:1548kB pagetables:8344kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 17 00:50:42 Server kernel: lowmem_reserve[]: 0 0 0 0
Jan 17 00:50:42 Server kernel: DMA: 364*4kB (UMEC) 342*8kB (UMEC) 149*16kB (UMEC) 75*32kB (UMEC) 21*64kB (UMEC) 12*128kB (UEC) 3*256kB (UMC) 2*512kB (ME) 1*1024kB (E) 1*2048kB (M) 0*4096kB = 16720kB
Jan 17 00:50:42 Server kernel: DMA32: 505*4kB (UME) 492*8kB (UME) 181*16kB (UME) 65*32kB (UME) 30*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12852kB
Jan 17 00:50:42 Server kernel: 289750 total pagecache pages
Jan 17 00:50:42 Server kernel: 0 pages in swap cache
Jan 17 00:50:42 Server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 17 00:50:42 Server kernel: Free swap = 0kB
Jan 17 00:50:42 Server kernel: Total swap = 0kB
Jan 17 00:50:42 Server kernel: 1025536 pages RAM
Jan 17 00:50:42 Server kernel: 0 pages HighMem/MovableOnly
Jan 17 00:50:42 Server kernel: 41235 pages reserved
Jan 17 00:50:42 Server kernel: 16384 pages cma reserved
Jan 17 00:50:42 Server kernel: Tasks state (memory values in pages):
Jan 17 00:50:42 Server kernel: [pid] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 17 00:50:42 Server kernel: [134] 0 134 6113 553 65536 0 -250 systemd-journal
Jan 17 00:50:42 Server kernel: [184] 0 184 5017 283 49152 0 -1000 systemd-udevd
Jan 17 00:50:42 Server kernel: [301] 0 301 884 373 40960 0 0 vmtouch
Jan 17 00:50:42 Server kernel: [313] 0 313 749 25 32768 0 0 rngd
Jan 17 00:50:42 Server kernel: [336] 0 336 1074 46 32768 0 0 dropbear
Jan 17 00:50:42 Server kernel: [405] 0 405 24837 332 65536 0 0 dhclient
Jan 17 00:50:42 Server kernel: [474] 0 474 3107 967 49152 0 0 dietpi-dashboar
Jan 17 00:50:42 Server kernel: [480] 106 480 1477638 597574 5484544 0 0 java
Jan 17 00:50:42 Server kernel: [487] 0 487 1017 26 32768 0 0 agetty
Jan 17 00:50:42 Server kernel: [490] 0 490 1017 27 40960 0 0 agetty
Jan 17 00:50:42 Server kernel: [636] 101 636 22529 710 73728 0 0 systemd-timesyn
Jan 17 00:50:42 Server kernel: [698] 0 698 18129 597 122880 0 0 nmbd
Jan 17 00:50:42 Server kernel: [737] 0 737 22096 1168 155648 0 0 smbd
Jan 17 00:50:42 Server kernel: [739] 0 739 21615 735 147456 0 0 smbd-notifyd
Jan 17 00:50:42 Server kernel: [740] 0 740 21617 737 143360 0 0 cleanupd
Jan 17 00:50:42 Server kernel: [744] 105 744 15628 591 102400 0 0 redis-server
Jan 17 00:50:42 Server kernel: [825] 104 825 454784 49139 630784 0 0 mariadbd
Jan 17 00:50:42 Server kernel: [858] 0 858 88563 2119 172032 0 0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [928] 33 928 2455 661 53248 0 0 lighttpd
Jan 17 00:50:42 Server kernel: [934] 0 934 1413 63 40960 0 0 cron
Jan 17 00:50:42 Server kernel: [941] 106 941 25951 139 69632 0 0 controller
Jan 17 00:50:42 Server kernel: [23222] 33 23222 114889 9336 262144 0 0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [25388] 33 25388 114397 8882 258048 0 0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [26025] 33 26025 88563 1467 159744 0 0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [26242] 33 26242 88563 1467 159744 0 0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [26272] 0 26272 2228 115 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26273] 33 26273 517 17 32768 0 0 sh
Jan 17 00:50:42 Server kernel: [26275] 33 26275 63974 10132 233472 0 0 php
Jan 17 00:50:42 Server kernel: [26292] 0 26292 2136 89 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26297] 0 26297 2136 89 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26298] 0 26298 2136 89 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26301] 0 26301 2136 89 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26304] 0 26304 2136 89 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26308] 0 26308 2136 88 49152 0 0 cron
Jan 17 00:50:42 Server kernel: [26309] 0 26309 22128 761 139264 0 0 smbd
Jan 17 00:50:42 Server kernel: [26312] 0 26312 1965 76 45056 0 0 cron
Jan 17 00:50:42 Server kernel: [26314] 0 26314 18129 597 118784 0 0 nmbd
Jan 17 00:50:42 Server kernel: [26319] 0 26319 1731 74 45056 0 0 cron
Jan 17 00:50:42 Server kernel: [26324] 0 26324 1637 71 45056 0 0 cron
Jan 17 00:50:42 Server kernel: [26326] 0 26326 1461 65 40960 0 0 cron
Jan 17 00:50:42 Server kernel: [26328] 0 26328 1455 65 40960 0 0 cron
Jan 17 00:50:42 Server kernel: [26331] 0 26331 1413 63 40960 0 0 cron
Jan 17 00:50:42 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=480,uid=106
Jan 17 00:50:42 Server kernel: Out of memory: Killed process 480 (java) total-vm:5910552kB, anon-rss:2390104kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5356kB oom_score_adj:0
Jan 17 00:50:42 Server kernel: oom_reaper: reaped process 480 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Unit process 941 (controller) remains running after unit stopped.
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Consumed 17min 11.069s CPU time.
Jan 17 00:50:42 Server systemd[1]: Starting Clean php session files...
Jan 17 00:50:42 Server CRON[26326]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26312]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26328]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26292]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26331]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26298]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26319]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26308]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26304]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26297]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26301]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26324]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26337]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26342]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26345]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26348]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26340]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 00:50:42 Server CRON[26349]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26343]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26341]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26346]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26347]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26350]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 17 00:50:42 Server CRON[26344]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26351]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 17 00:50:42 Server CRON[26339]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26326]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:42 Server CRON[26297]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:43 Server CRON[26304]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:44 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 17 00:50:44 Server systemd[1]: Finished Clean php session files.
Jan 17 00:50:47 Server CRON[26308]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26324]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26337]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26331]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26312]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26292]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26319]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26298]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26328]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:49 Server CRON[26301]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:53:01 Server CRON[26272]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:55:22 Server CRON[26521]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:55:22 Server CRON[26522]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:55:23 Server CRON[26521]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:55:59 Server systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
Jan 17 00:56:01 Server fstrim[26530]: /boot: 91.6 MiB (96023040 bytes) trimmed on /dev/mmcblk0p1
Jan 17 00:56:01 Server fstrim[26530]: /: 55.2 GiB (59294412800 bytes) trimmed on /dev/mmcblk0p2
Jan 17 00:56:01 Server systemd[1]: fstrim.service: Succeeded.
Jan 17 00:56:01 Server systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
Jan 17 01:00:06 Server CRON[26603]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:00:06 Server CRON[26604]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:00:07 Server CRON[26603]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:05:07 Server CRON[26609]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:05:07 Server CRON[26610]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:05:23 Server systemd[1]: Starting Certbot...
Jan 17 01:05:25 Server systemd[1]: certbot.service: Succeeded.
Jan 17 01:05:25 Server systemd[1]: Finished Certbot.
Jan 17 01:05:25 Server systemd[1]: certbot.service: Consumed 1.311s CPU time.
Jan 17 01:07:55 Server CRON[26609]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:09:01 Server CRON[26640]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:09:01 Server CRON[26641]: (root) CMD ( [-x /usr/lib/php/sessionclean] && if [! -d /run/systemd/system]; then /usr/lib/php/sessionclean; fi)
Jan 17 01:09:01 Server CRON[26640]: pam_unix(cron:session): session closed for user root
Jan 17 01:09:01 Server systemd[1]: Starting Clean php session files...
Jan 17 01:09:01 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 17 01:09:01 Server systemd[1]: Finished Clean php session files.
Jan 17 01:10:19 Server CRON[26690]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:10:19 Server CRON[26691]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:10:21 Server CRON[26690]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:15:12 Server CRON[26770]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:15:12 Server CRON[26771]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:15:13 Server CRON[26770]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:17:12 Server CRON[26776]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:17:12 Server CRON[26777]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 01:17:12 Server CRON[26776]: pam_unix(cron:session): session closed for user root
Jan 17 01:20:12 Server CRON[26862]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:20:12 Server CRON[26863]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:20:14 Server CRON[26862]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:25:12 Server CRON[26871]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:25:12 Server CRON[26872]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:12 Server CRON[26873]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:25:12 Server CRON[26874]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Jan 17 01:25:12 Server systemd[1]: Stopping Network Time Synchronization...
Jan 17 01:25:12 Server systemd[1]: systemd-timesyncd.service: Succeeded.
Jan 17 01:25:12 Server systemd[1]: Stopped Network Time Synchronization.
Jan 17 01:25:12 Server systemd[1]: systemd-timesyncd.service: Consumed 37.432s CPU time.
Jan 17 01:25:13 Server runuser[27028]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:13 Server runuser[27028]: pam_unix(runuser:session): session closed for user www-data
Jan 17 01:25:13 Server runuser[27032]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:13 Server runuser[27032]: pam_unix(runuser:session): session closed for user www-data
Jan 17 01:25:13 Server CRON[26871]: pam_unix(cron:session): session closed for user root
Jan 17 01:25:13 Server CRON[26872]: pam_unix(cron:session): session closed for user www-data
J

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 17, 2022, 10:58pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/9 "2022-01-17T22:58:41Z")

</div>

This indicates that as well as Elasticsearch you have MariaDB, Redis, and several heavy PHP processes all running on this machine. It just doesn't have enough RAM to cope. The advice from my previous message still stands:

> [@DavidTurner](#):
>
> Either use a smaller heap or reduce the needs of the other things that this machine is running.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2022, 10:59pm UTC](https://discuss.elastic.co/t/every-night-elasticsearch-code-killed-signal-kill/294397/10 "2022-02-14T22:59:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.