Every Night- Elasticsearch. (code=killed, signal=KILL)

Here the system log from the time Elasticsearch stopped.

journalctl --since "2022-01-15 01:30" --until "2022-01-15 02:00"

Jan 15 01:52:27 Server kernel: rpimonitord invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 15 01:52:27 Server kernel: CPU: 2 PID: 904 Comm: rpimonitord Not tainted 5.10.63-v8+ #1488
Jan 15 01:52:27 Server kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jan 15 01:52:27 Server kernel: Call trace:
Jan 15 01:52:27 Server kernel:  dump_backtrace+0x0/0x1b8
Jan 15 01:52:27 Server kernel:  show_stack+0x20/0x30
Jan 15 01:52:27 Server kernel:  dump_stack+0xf0/0x158
Jan 15 01:52:27 Server kernel:  dump_header+0x4c/0x204
Jan 15 01:52:27 Server kernel:  oom_kill_process+0x1f8/0x200
Jan 15 01:52:27 Server kernel:  out_of_memory+0xf0/0x330
Jan 15 01:52:27 Server kernel:  __alloc_pages_slowpath.constprop.133+0x820/0xb80
Jan 15 01:52:27 Server kernel:  __alloc_pages_nodemask+0x2bc/0x330
Jan 15 01:52:27 Server kernel:  pagecache_get_page+0x144/0x2e0
Jan 15 01:52:27 Server kernel:  filemap_fault+0x520/0x9e0
Jan 15 01:52:27 Server kernel:  ext4_filemap_fault+0x3c/0x818
Jan 15 01:52:27 Server kernel:  __do_fault+0x44/0x110
Jan 15 01:52:27 Server kernel:  handle_mm_fault+0x6a8/0xcb8
Jan 15 01:52:27 Server kernel:  do_page_fault+0x148/0x3e8
Jan 15 01:52:27 Server kernel:  do_translation_fault+0x58/0x70
Jan 15 01:52:27 Server kernel:  do_mem_abort+0x48/0xa8
Jan 15 01:52:27 Server kernel:  el0_ia+0x68/0xc8
Jan 15 01:52:27 Server kernel:  el0_sync_handler+0x78/0xb8
Jan 15 01:52:27 Server kernel:  el0_sync+0x180/0x1c0
Jan 15 01:52:27 Server kernel: Mem-Info:
Jan 15 01:52:27 Server kernel: active_anon:263544 inactive_anon:694226 isolated_anon:0
                                active_file:231 inactive_file:0 isolated_file:0
                                unevictable:387 dirty:0 writeback:0
                                slab_reclaimable:5894 slab_unreclaimable:5440
                                mapped:7320 shmem:275152 pagetables:2615 bounce:0
                                free:6987 free_pcp:245 free_cma:468
Jan 15 01:52:27 Server kernel: Node 0 active_anon:1054176kB inactive_anon:2776904kB active_file:924kB inactive_file:0kB unevictable:1548kB isolated(anon):0kB isolated(file):0kB mapped:29280kB dirty:0kB writeback:0kB shmem:1100608kB writeback_tmp:0kB kernel_stack:3664kB all_unreclaimable? no
Jan 15 01:52:27 Server kernel: DMA free:15576kB min:3900kB low:4872kB high:5844kB reserved_highatomic:0KB active_anon:600340kB inactive_anon:306160kB active_file:0kB inactive_file:220kB unevictable:0kB writepending:0kB present:1021952kB managed:937872kB mlocked:0kB pagetables:1768kB bounce:0kB free_pcp:960kB local_pcp:280kB free_cma:1872kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 2929 2929 2929
Jan 15 01:52:27 Server kernel: DMA32 free:12372kB min:12480kB low:15600kB high:18720kB reserved_highatomic:0KB active_anon:453836kB inactive_anon:2470744kB active_file:0kB inactive_file:284kB unevictable:1548kB writepending:0kB present:3080192kB managed:2999332kB mlocked:1548kB pagetables:8692kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 0 0 0
Jan 15 01:52:27 Server kernel: DMA: 592*4kB (UMEC) 327*8kB (UMEC) 129*16kB (UEC) 102*32kB (UEC) 44*64kB (UEC) 10*128kB (UEC) 0*256kB 3*512kB (C) 0*1024kB 0*2048kB 0*4096kB = 15944kB
Jan 15 01:52:27 Server kernel: DMA32: 409*4kB (UME) 548*8kB (UME) 208*16kB (UME) 43*32kB (UM) 30*64kB (UME) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12900kB
Jan 15 01:52:27 Server kernel: 275866 total pagecache pages
Jan 15 01:52:27 Server kernel: 0 pages in swap cache
Jan 15 01:52:27 Server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 15 01:52:27 Server kernel: Free swap  = 0kB
Jan 15 01:52:27 Server kernel: Total swap = 0kB
Jan 15 01:52:27 Server kernel: 1025536 pages RAM
Jan 15 01:52:27 Server kernel: 0 pages HighMem/MovableOnly
Jan 15 01:52:27 Server kernel: 41235 pages reserved
Jan 15 01:52:27 Server kernel: 16384 pages cma reserved
Jan 15 01:52:27 Server kernel: Tasks state (memory values in pages):
Jan 15 01:52:27 Server kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 15 01:52:27 Server kernel: [    137]     0   137     6115      597    73728        0          -250 systemd-journal
Jan 15 01:52:27 Server kernel: [    185]     0   185     5016      282    49152        0         -1000 systemd-udevd
Jan 15 01:52:27 Server kernel: [    306]     0   306      884      358    32768        0             0 vmtouch
Jan 15 01:52:27 Server kernel: [    313]     0   313      749       26    36864        0             0 rngd
Jan 15 01:52:27 Server kernel: [    334]     0   334     1074       48    36864        0             0 dropbear
Jan 15 01:52:27 Server kernel: [    405]     0   405    24837      325    61440        0             0 dhclient
Jan 15 01:52:27 Server kernel: [    471]   101   471    22661      850    73728        0             0 systemd-timesyn
Jan 15 01:52:27 Server kernel: [    492]     0   492     3107      966    45056        0             0 dietpi-dashboar
Jan 15 01:52:27 Server kernel: [    502]     0   502     1017       26    36864        0             0 agetty
Jan 15 01:52:27 Server kernel: [    503]     0   503     1017       28    40960        0             0 agetty
Jan 15 01:52:27 Server kernel: [    673]     0   673    18129      596   118784        0             0 nmbd
Jan 15 01:52:27 Server kernel: [    717]     0   717    22096     1168   155648        0             0 smbd
Jan 15 01:52:27 Server kernel: [    719]     0   719    21615      735   147456        0             0 smbd-notifyd
Jan 15 01:52:27 Server kernel: [    720]     0   720    21617      737   143360        0             0 cleanupd
Jan 15 01:52:27 Server kernel: [    723]   105   723    15628      592   102400        0             0 redis-server
Jan 15 01:52:27 Server kernel: [    781]   104   781   454773    49713   626688        0             0 mariadbd
Jan 15 01:52:27 Server kernel: [    823]     0   823    88563     2119   176128        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [    880]    33   880     2480      659    49152        0             0 lighttpd
Jan 15 01:52:27 Server kernel: [    902]     0   902    18217     4060   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    903]  1000   903    18349     4198   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    904]     0   904    18264     4069   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    914]     0   914     1413       64    40960        0             0 cron
Jan 15 01:52:27 Server kernel: [   2020]   106  2020  1471905   600355  5423104        0             0 java
Jan 15 01:52:27 Server kernel: [   2221]   106  2221    25951      142    77824        0             0 controller
Jan 15 01:52:27 Server kernel: [   5386]     0  5386     1140       96    36864        0             0 dropbear
Jan 15 01:52:27 Server kernel: [  21156]     0 21156     1949      625    49152        0             0 bash
Jan 15 01:52:27 Server kernel: [  21187]     0 21187     3552      422    61440        0             0 mc
Jan 15 01:52:27 Server kernel: [  21189]     0 21189     1948      625    40960        0             0 bash
Jan 15 01:52:27 Server kernel: [  21501]     0 21501     2014      694    49152        0             0 bash
Jan 15 01:52:27 Server kernel: [  24723]    33 24723    89797     8935   233472        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24774]    33 24774   114377     8663   249856        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24838]    33 24838    89776     8635   225280        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24854]    33 24854    89285     8123   221184        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  57209]     0 57209     2228      116    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57210]    33 57210      517       17    36864        0             0 sh
Jan 15 01:52:27 Server kernel: [  57211]    33 57211    63301     9284   225280        0             0 php
Jan 15 01:52:27 Server kernel: [  57243]     0 57243     2136      105    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57251]     0 57251     2136      105    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57252]     0 57252     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57256]     0 57256     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57258]     0 57258     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57264]     0 57264     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57271]     0 57271     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57275]     0 57275     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57278]     0 57278     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57280]     0 57280     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57285]     0 57285     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57288]     0 57288     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57290]     0 57290     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57295]     0 57295     2136       89    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57296]     0 57296     2136       89    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57301]     0 57301     2027       79    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57304]     0 57304    22136      760   139264        0             0 smbd
Jan 15 01:52:27 Server kernel: [  57305]     0 57305     1637       72    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57309]     0 57309     1534       68    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57310]     0 57310     1534       68    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57314]     0 57314    18129      596   114688        0             0 nmbd
Jan 15 01:52:27 Server kernel: [  57315]     0 57315     1413       64    40960        0             0 cron
Jan 15 01:52:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=2020,uid=106
Jan 15 01:52:27 Server kernel: Out of memory: Killed process 2020 (java) total-vm:5887620kB, anon-rss:2401228kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5296kB oom_score_adj:0
Jan 15 01:52:27 Server kernel: oom_reaper: reaped process 2020 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Unit process 2221 (controller) remains running after unit stopped.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Consumed 22min 47.135s CPU time.
Jan 15 01:52:23 Server systemd[1]: Starting Clean php session files...
Jan 15 01:52:23 Server CRON[57252]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57295]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57271]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57310]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57305]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57288]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57315]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57258]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57296]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57301]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57264]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57319]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57280]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57275]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57243]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57256]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57290]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57285]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57325]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57328]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57327]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57326]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Jan 15 01:52:23 Server CRON[57335]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57324]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57336]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57330]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57333]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57337]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57329]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57332]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57334]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57331]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57338]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57339]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57340]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57341]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57342]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 15 01:52:23 Server CRON[57344]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57343]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:24 Server systemd[1]: Stopping Network Time Synchronization...
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Succeeded.
Jan 15 01:52:24 Server systemd[1]: Stopped Network Time Synchronization.
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Consumed 48.219s CPU time.
Jan 15 01:52:24 Server CRON[57288]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:25 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 01:52:25 Server systemd[1]: Finished Clean php session files.
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:25 Server runuser[57574]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:26 Server runuser[57574]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:26 Server CRON[57295]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:29 Server CRON[57252]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57285]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57310]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57319]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57271]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57296]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57275]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57301]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57305]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57315]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57256]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57290]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57258]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57243]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57280]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:55:02 Server CRON[57710]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:55:02 Server CRON[57711]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:55:03 Server CRON[57710]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:56:25 Server CRON[57209]: pam_unix(cron:session): session closed for user www-data