Every Night- Elasticsearch. (code=killed, signal=KILL)

Elastic Stack Elasticsearch
1

Hey,

on my Rasperry Pi with Nextcloud and full text search (powered by Elasticsearch) every night
Elasticsearch is killed (code=killed, signal=KILL).

ā— elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2022-01-13 01:27:21 CET; 1 day 8h ago
       Docs: https://www.elastic.co
    Process: 670366 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
   Main PID: 670366 (code=killed, signal=KILL)
        CPU: 20min 11.324s

Jan 12 07:49:17 Server systemd[1]: Starting Elasticsearch...
Jan 12 07:50:13 Server systemd[1]: Started Elasticsearch.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Unit process 670588 (controller) remains running after unit stopped.
Jan 13 01:27:21 Server systemd[1]: elasticsearch.service: Consumed 20min 11.308s CPU time.

Here the gc.log.23

[2022-01-14T10:20:17.766+0000][2176][gc] Using G1
[2022-01-14T10:20:17.808+0000][2176][gc,init] Version: 17.0.1+12 (release)
[2022-01-14T10:20:17.808+0000][2176][gc,init] CPUs: 4 total, 4 available
[2022-01-14T10:20:17.808+0000][2176][gc,init] Memory: 3844M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Large Page Support: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] NUMA Support: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] Compressed Oops: Enabled (32-bit)
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Region Size: 1M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Min Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Initial Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Heap Max Capacity: 1922M
[2022-01-14T10:20:17.808+0000][2176][gc,init] Pre-touch: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,init] Parallel Workers: 4
[2022-01-14T10:20:17.808+0000][2176][gc,init] Concurrent Workers: 1
[2022-01-14T10:20:17.808+0000][2176][gc,init] Concurrent Refinement Workers: 4
[2022-01-14T10:20:17.808+0000][2176][gc,init] Periodic GC: Disabled
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] CDS archive(s) not mapped
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] Compressed class space mapped at: 0x0000000100000000-0x0000000140000000, reserved size: 1073741824
[2022-01-14T10:20:17.808+0000][2176][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x140000000
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit] Heap
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit]  garbage-first heap   total 1968128K, used 1538K [0x0000000087e00000, 0x0000000100000000)
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit]   region size 1024K, 2 young (2048K), 0 survivors (0K)
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit]  Metaspace       used 3427K, committed 3520K, reserved 1056768K
[2022-01-14T10:20:18.014+0000][2176][gc,heap,exit]   class space    used 269K, committed 320K, reserved 1048576K

Any idea?
Greetings
beohna

My environment:
Rasperry Pi - model : RPi 4 Model B (aarch64) 4GB RAM
DietPi v7.9.3
Nextcloud v23

2

You need to share Elasticsearch logs and also system logs from the time Elasticsearch stopped.

3

Here the system log from the time Elasticsearch stopped.

journalctl --since "2022-01-15 01:30" --until "2022-01-15 02:00"

Jan 15 01:52:27 Server kernel: rpimonitord invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 15 01:52:27 Server kernel: CPU: 2 PID: 904 Comm: rpimonitord Not tainted 5.10.63-v8+ #1488
Jan 15 01:52:27 Server kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jan 15 01:52:27 Server kernel: Call trace:
Jan 15 01:52:27 Server kernel:  dump_backtrace+0x0/0x1b8
Jan 15 01:52:27 Server kernel:  show_stack+0x20/0x30
Jan 15 01:52:27 Server kernel:  dump_stack+0xf0/0x158
Jan 15 01:52:27 Server kernel:  dump_header+0x4c/0x204
Jan 15 01:52:27 Server kernel:  oom_kill_process+0x1f8/0x200
Jan 15 01:52:27 Server kernel:  out_of_memory+0xf0/0x330
Jan 15 01:52:27 Server kernel:  __alloc_pages_slowpath.constprop.133+0x820/0xb80
Jan 15 01:52:27 Server kernel:  __alloc_pages_nodemask+0x2bc/0x330
Jan 15 01:52:27 Server kernel:  pagecache_get_page+0x144/0x2e0
Jan 15 01:52:27 Server kernel:  filemap_fault+0x520/0x9e0
Jan 15 01:52:27 Server kernel:  ext4_filemap_fault+0x3c/0x818
Jan 15 01:52:27 Server kernel:  __do_fault+0x44/0x110
Jan 15 01:52:27 Server kernel:  handle_mm_fault+0x6a8/0xcb8
Jan 15 01:52:27 Server kernel:  do_page_fault+0x148/0x3e8
Jan 15 01:52:27 Server kernel:  do_translation_fault+0x58/0x70
Jan 15 01:52:27 Server kernel:  do_mem_abort+0x48/0xa8
Jan 15 01:52:27 Server kernel:  el0_ia+0x68/0xc8
Jan 15 01:52:27 Server kernel:  el0_sync_handler+0x78/0xb8
Jan 15 01:52:27 Server kernel:  el0_sync+0x180/0x1c0
Jan 15 01:52:27 Server kernel: Mem-Info:
Jan 15 01:52:27 Server kernel: active_anon:263544 inactive_anon:694226 isolated_anon:0
                                active_file:231 inactive_file:0 isolated_file:0
                                unevictable:387 dirty:0 writeback:0
                                slab_reclaimable:5894 slab_unreclaimable:5440
                                mapped:7320 shmem:275152 pagetables:2615 bounce:0
                                free:6987 free_pcp:245 free_cma:468
Jan 15 01:52:27 Server kernel: Node 0 active_anon:1054176kB inactive_anon:2776904kB active_file:924kB inactive_file:0kB unevictable:1548kB isolated(anon):0kB isolated(file):0kB mapped:29280kB dirty:0kB writeback:0kB shmem:1100608kB writeback_tmp:0kB kernel_stack:3664kB all_unreclaimable? no
Jan 15 01:52:27 Server kernel: DMA free:15576kB min:3900kB low:4872kB high:5844kB reserved_highatomic:0KB active_anon:600340kB inactive_anon:306160kB active_file:0kB inactive_file:220kB unevictable:0kB writepending:0kB present:1021952kB managed:937872kB mlocked:0kB pagetables:1768kB bounce:0kB free_pcp:960kB local_pcp:280kB free_cma:1872kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 2929 2929 2929
Jan 15 01:52:27 Server kernel: DMA32 free:12372kB min:12480kB low:15600kB high:18720kB reserved_highatomic:0KB active_anon:453836kB inactive_anon:2470744kB active_file:0kB inactive_file:284kB unevictable:1548kB writepending:0kB present:3080192kB managed:2999332kB mlocked:1548kB pagetables:8692kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB
Jan 15 01:52:27 Server kernel: lowmem_reserve[]: 0 0 0 0
Jan 15 01:52:27 Server kernel: DMA: 592*4kB (UMEC) 327*8kB (UMEC) 129*16kB (UEC) 102*32kB (UEC) 44*64kB (UEC) 10*128kB (UEC) 0*256kB 3*512kB (C) 0*1024kB 0*2048kB 0*4096kB = 15944kB
Jan 15 01:52:27 Server kernel: DMA32: 409*4kB (UME) 548*8kB (UME) 208*16kB (UME) 43*32kB (UM) 30*64kB (UME) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12900kB
Jan 15 01:52:27 Server kernel: 275866 total pagecache pages
Jan 15 01:52:27 Server kernel: 0 pages in swap cache
Jan 15 01:52:27 Server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 15 01:52:27 Server kernel: Free swap  = 0kB
Jan 15 01:52:27 Server kernel: Total swap = 0kB
Jan 15 01:52:27 Server kernel: 1025536 pages RAM
Jan 15 01:52:27 Server kernel: 0 pages HighMem/MovableOnly
Jan 15 01:52:27 Server kernel: 41235 pages reserved
Jan 15 01:52:27 Server kernel: 16384 pages cma reserved
Jan 15 01:52:27 Server kernel: Tasks state (memory values in pages):
Jan 15 01:52:27 Server kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 15 01:52:27 Server kernel: [    137]     0   137     6115      597    73728        0          -250 systemd-journal
Jan 15 01:52:27 Server kernel: [    185]     0   185     5016      282    49152        0         -1000 systemd-udevd
Jan 15 01:52:27 Server kernel: [    306]     0   306      884      358    32768        0             0 vmtouch
Jan 15 01:52:27 Server kernel: [    313]     0   313      749       26    36864        0             0 rngd
Jan 15 01:52:27 Server kernel: [    334]     0   334     1074       48    36864        0             0 dropbear
Jan 15 01:52:27 Server kernel: [    405]     0   405    24837      325    61440        0             0 dhclient
Jan 15 01:52:27 Server kernel: [    471]   101   471    22661      850    73728        0             0 systemd-timesyn
Jan 15 01:52:27 Server kernel: [    492]     0   492     3107      966    45056        0             0 dietpi-dashboar
Jan 15 01:52:27 Server kernel: [    502]     0   502     1017       26    36864        0             0 agetty
Jan 15 01:52:27 Server kernel: [    503]     0   503     1017       28    40960        0             0 agetty
Jan 15 01:52:27 Server kernel: [    673]     0   673    18129      596   118784        0             0 nmbd
Jan 15 01:52:27 Server kernel: [    717]     0   717    22096     1168   155648        0             0 smbd
Jan 15 01:52:27 Server kernel: [    719]     0   719    21615      735   147456        0             0 smbd-notifyd
Jan 15 01:52:27 Server kernel: [    720]     0   720    21617      737   143360        0             0 cleanupd
Jan 15 01:52:27 Server kernel: [    723]   105   723    15628      592   102400        0             0 redis-server
Jan 15 01:52:27 Server kernel: [    781]   104   781   454773    49713   626688        0             0 mariadbd
Jan 15 01:52:27 Server kernel: [    823]     0   823    88563     2119   176128        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [    880]    33   880     2480      659    49152        0             0 lighttpd
Jan 15 01:52:27 Server kernel: [    902]     0   902    18217     4060   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    903]  1000   903    18349     4198   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    904]     0   904    18264     4069   126976        0             0 rpimonitord
Jan 15 01:52:27 Server kernel: [    914]     0   914     1413       64    40960        0             0 cron
Jan 15 01:52:27 Server kernel: [   2020]   106  2020  1471905   600355  5423104        0             0 java
Jan 15 01:52:27 Server kernel: [   2221]   106  2221    25951      142    77824        0             0 controller
Jan 15 01:52:27 Server kernel: [   5386]     0  5386     1140       96    36864        0             0 dropbear
Jan 15 01:52:27 Server kernel: [  21156]     0 21156     1949      625    49152        0             0 bash
Jan 15 01:52:27 Server kernel: [  21187]     0 21187     3552      422    61440        0             0 mc
Jan 15 01:52:27 Server kernel: [  21189]     0 21189     1948      625    40960        0             0 bash
Jan 15 01:52:27 Server kernel: [  21501]     0 21501     2014      694    49152        0             0 bash
Jan 15 01:52:27 Server kernel: [  24723]    33 24723    89797     8935   233472        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24774]    33 24774   114377     8663   249856        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24838]    33 24838    89776     8635   225280        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  24854]    33 24854    89285     8123   221184        0             0 php-fpm7.4
Jan 15 01:52:27 Server kernel: [  57209]     0 57209     2228      116    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57210]    33 57210      517       17    36864        0             0 sh
Jan 15 01:52:27 Server kernel: [  57211]    33 57211    63301     9284   225280        0             0 php
Jan 15 01:52:27 Server kernel: [  57243]     0 57243     2136      105    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57251]     0 57251     2136      105    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57252]     0 57252     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57256]     0 57256     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57258]     0 57258     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57264]     0 57264     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57271]     0 57271     2136      103    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57275]     0 57275     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57278]     0 57278     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57280]     0 57280     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57285]     0 57285     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57288]     0 57288     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57290]     0 57290     2136       90    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57295]     0 57295     2136       89    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57296]     0 57296     2136       89    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57301]     0 57301     2027       79    49152        0             0 cron
Jan 15 01:52:27 Server kernel: [  57304]     0 57304    22136      760   139264        0             0 smbd
Jan 15 01:52:27 Server kernel: [  57305]     0 57305     1637       72    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57309]     0 57309     1534       68    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57310]     0 57310     1534       68    45056        0             0 cron
Jan 15 01:52:27 Server kernel: [  57314]     0 57314    18129      596   114688        0             0 nmbd
Jan 15 01:52:27 Server kernel: [  57315]     0 57315     1413       64    40960        0             0 cron
Jan 15 01:52:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=2020,uid=106
Jan 15 01:52:27 Server kernel: Out of memory: Killed process 2020 (java) total-vm:5887620kB, anon-rss:2401228kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5296kB oom_score_adj:0
Jan 15 01:52:27 Server kernel: oom_reaper: reaped process 2020 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Unit process 2221 (controller) remains running after unit stopped.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Consumed 22min 47.135s CPU time.
Jan 15 01:52:23 Server systemd[1]: Starting Clean php session files...
Jan 15 01:52:23 Server CRON[57252]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57295]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57271]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57310]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57305]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57288]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57315]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57258]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57296]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57301]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57264]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57319]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57280]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57275]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57243]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57256]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57290]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57285]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 15 01:52:23 Server CRON[57325]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57328]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57327]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57326]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Jan 15 01:52:23 Server CRON[57335]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57324]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57336]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57330]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57333]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57337]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57329]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57332]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57334]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57331]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 01:52:23 Server CRON[57338]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57339]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57340]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57341]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57342]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 15 01:52:23 Server CRON[57344]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57343]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:52:23 Server CRON[57309]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57251]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:23 Server CRON[57278]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:24 Server systemd[1]: Stopping Network Time Synchronization...
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Succeeded.
Jan 15 01:52:24 Server systemd[1]: Stopped Network Time Synchronization.
Jan 15 01:52:24 Server systemd[1]: systemd-timesyncd.service: Consumed 48.219s CPU time.
Jan 15 01:52:24 Server CRON[57288]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:25 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 01:52:25 Server systemd[1]: Finished Clean php session files.
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:25 Server runuser[57570]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:25 Server runuser[57574]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:52:26 Server runuser[57574]: pam_unix(runuser:session): session closed for user www-data
Jan 15 01:52:26 Server CRON[57295]: pam_unix(cron:session): session closed for user root
Jan 15 01:52:29 Server CRON[57252]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57285]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57310]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57319]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57271]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57296]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57275]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57301]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57305]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57315]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57256]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:29 Server CRON[57290]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57258]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57243]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:52:30 Server CRON[57280]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:55:02 Server CRON[57710]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 15 01:55:02 Server CRON[57711]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 15 01:55:03 Server CRON[57710]: pam_unix(cron:session): session closed for user www-data
Jan 15 01:56:25 Server CRON[57209]: pam_unix(cron:session): session closed for user www-data
4

service Elasticsearch status

ā— elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Sat 2022-01-15 01:52:23 CET; 14h ago
       Docs: https://www.elastic.co
    Process: 2020 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
   Main PID: 2020 (code=killed, signal=KILL)
        CPU: 22min 47.145s

Jan 14 11:20:14 Server systemd[1]: Starting Elasticsearch...
Jan 14 11:21:14 Server systemd[1]: Started Elasticsearch.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Unit process 2221 (controller) remains running after unit stopped.
Jan 15 01:52:23 Server systemd[1]: elasticsearch.service: Consumed 22min 47.135s CPU time.

And sorry the Elasticsearch.log is empty.

Greetings
beohna

5

The process is being killed by Out of Memory reasons:

Jan 15 01:52:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=2020,uid=106
Jan 15 01:52:27 Server kernel: Out of memory: Killed process 2020 (java) total-vm:5887620kB, anon-rss:2401228kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5296kB oom_score_adj:0
Jan 15 01:52:27 Server kernel: oom_reaper: reaped process 2020 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB

These OOM issues can happen for many reasons, one is having too much data for a small node and make expensive queries.

Normally the solution would be to increase the memory of the server and the service, but since you are running with limited resources on a Raspberry Pi, you can't do that.

What you can try to do is to check the number of indices and shards you have in the node and see if you can reduce it to make your single-node cluster more stable, but I do not know how Nextcloud uses Elasticsearch.

You can check the number of shards using the following curl request from your node:

curl -X GET "localhost:9200/_cat/shards?pretty"

Can you run that and share the result?

6

Note that this is the OS oom-killer, not a JVM OutOfMemoryException so the problem is not that Elasticsearch ran out of heap (too much data etc) it's that Elasticsearch plus everything else on the same machine consumed too much memory. Either use a smaller heap or reduce the needs of the other things that this machine is running.

1 Like
7

curl -X GET "localhost:9200/_cat/shards?pretty":

.ds-ilm-history-5-2022.01.12-000002                           0 p STARTED                  127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.28-000002 0 p STARTED                  127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.28-000002 0 r UNASSIGNED
.geoip_databases                                              0 p STARTED       42  52.2mb 127.0.0.1 Server
.ds-ilm-history-5-2021.12.13-000001                           0 p STARTED                  127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2022.01.11-000003 0 p STARTED                  127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2022.01.11-000003 0 r UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 p STARTED                  127.0.0.1 Server
.ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 r UNASSIGNED
cloud.XXXXXX.com                                             0 p STARTED    44823 163.6mb 127.0.0.1 Server
cloud.XXXXXX.com                                             0 r UNASSIGNED

This is the result after a restart of the Elasticsearch service.

I also uninstall the RPi-Monitor, I think this can be the reason of the memory problem.

Thanks for the help.

8

next night...

journalctl --since "2022-01-17 00:30" --until "2022-01-17 02:00"

Jan 17 00:50:41 Server kernel: php invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 17 00:50:42 Server kernel: CPU: 2 PID: 26275 Comm: php Not tainted 5.10.63-v8+ #1488
Jan 17 00:50:42 Server kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jan 17 00:50:42 Server kernel: Call trace:
Jan 17 00:50:42 Server kernel:  dump_backtrace+0x0/0x1b8
Jan 17 00:50:42 Server kernel:  show_stack+0x20/0x30
Jan 17 00:50:42 Server kernel:  dump_stack+0xf0/0x158
Jan 17 00:50:42 Server kernel:  dump_header+0x4c/0x204
Jan 17 00:50:42 Server kernel:  oom_kill_process+0x1f8/0x200
Jan 17 00:50:42 Server kernel:  out_of_memory+0xf0/0x330
Jan 17 00:50:42 Server kernel:  __alloc_pages_slowpath.constprop.133+0x820/0xb80
Jan 17 00:50:42 Server kernel:  __alloc_pages_nodemask+0x2bc/0x330
Jan 17 00:50:42 Server kernel:  pagecache_get_page+0x144/0x2e0
Jan 17 00:50:42 Server kernel:  filemap_fault+0x520/0x9e0
Jan 17 00:50:42 Server kernel:  ext4_filemap_fault+0x3c/0x818
Jan 17 00:50:42 Server kernel:  __do_fault+0x44/0x110
Jan 17 00:50:42 Server kernel:  handle_mm_fault+0x6a8/0xcb8
Jan 17 00:50:42 Server kernel:  do_page_fault+0x148/0x3e8
Jan 17 00:50:42 Server kernel:  do_translation_fault+0x58/0x70
Jan 17 00:50:42 Server kernel:  do_mem_abort+0x48/0xa8
Jan 17 00:50:42 Server kernel:  el0_da+0x40/0x58
Jan 17 00:50:42 Server kernel:  el0_sync_handler+0x68/0xb8
Jan 17 00:50:42 Server kernel:  el0_sync+0x180/0x1c0
Jan 17 00:50:42 Server kernel: Mem-Info:
Jan 17 00:50:42 Server kernel: active_anon:277490 inactive_anon:680911 isolated_anon:0
                                active_file:170 inactive_file:0 isolated_file:0
                                unevictable:387 dirty:0 writeback:0
                                slab_reclaimable:5708 slab_unreclaimable:5063
                                mapped:7337 shmem:288966 pagetables:2359 bounce:0
                                free:7405 free_pcp:124 free_cma:386
Jan 17 00:50:42 Server kernel: Node 0 active_anon:1109960kB inactive_anon:2723644kB active_file:680kB inactive_file:0kB unevictable:1548kB isolated(anon):0kB isolated(file):0kB mapped:29348kB dirty:0kB writeback:0kB shmem:1155864kB writeback_tmp:0kB kernel_stack:3376kB all_unreclaimable? no
Jan 17 00:50:42 Server kernel: DMA free:17144kB min:3900kB low:4872kB high:5844kB reserved_highatomic:0KB active_anon:668892kB inactive_anon:240396kB active_file:0kB inactive_file:304kB unevictable:0kB writepending:0kB present:1021952kB managed:937872kB mlocked:0kB pagetables:1092kB bounce:0kB free_pcp:496kB local_pcp:248kB free_cma:1544kB
Jan 17 00:50:42 Server kernel: lowmem_reserve[]: 0 2929 2929 2929
Jan 17 00:50:42 Server kernel: DMA32 free:12476kB min:12480kB low:15600kB high:18720kB reserved_highatomic:0KB active_anon:441068kB inactive_anon:2483248kB active_file:0kB inactive_file:128kB unevictable:1548kB writepending:0kB present:3080192kB managed:2999332kB mlocked:1548kB pagetables:8344kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 17 00:50:42 Server kernel: lowmem_reserve[]: 0 0 0 0
Jan 17 00:50:42 Server kernel: DMA: 364*4kB (UMEC) 342*8kB (UMEC) 149*16kB (UMEC) 75*32kB (UMEC) 21*64kB (UMEC) 12*128kB (UEC) 3*256kB (UMC) 2*512kB (ME) 1*1024kB (E) 1*2048kB (M) 0*4096kB = 16720kB
Jan 17 00:50:42 Server kernel: DMA32: 505*4kB (UME) 492*8kB (UME) 181*16kB (UME) 65*32kB (UME) 30*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12852kB
Jan 17 00:50:42 Server kernel: 289750 total pagecache pages
Jan 17 00:50:42 Server kernel: 0 pages in swap cache
Jan 17 00:50:42 Server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 17 00:50:42 Server kernel: Free swap  = 0kB
Jan 17 00:50:42 Server kernel: Total swap = 0kB
Jan 17 00:50:42 Server kernel: 1025536 pages RAM
Jan 17 00:50:42 Server kernel: 0 pages HighMem/MovableOnly
Jan 17 00:50:42 Server kernel: 41235 pages reserved
Jan 17 00:50:42 Server kernel: 16384 pages cma reserved
Jan 17 00:50:42 Server kernel: Tasks state (memory values in pages):
Jan 17 00:50:42 Server kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 17 00:50:42 Server kernel: [    134]     0   134     6113      553    65536        0          -250 systemd-journal
Jan 17 00:50:42 Server kernel: [    184]     0   184     5017      283    49152        0         -1000 systemd-udevd
Jan 17 00:50:42 Server kernel: [    301]     0   301      884      373    40960        0             0 vmtouch
Jan 17 00:50:42 Server kernel: [    313]     0   313      749       25    32768        0             0 rngd
Jan 17 00:50:42 Server kernel: [    336]     0   336     1074       46    32768        0             0 dropbear
Jan 17 00:50:42 Server kernel: [    405]     0   405    24837      332    65536        0             0 dhclient
Jan 17 00:50:42 Server kernel: [    474]     0   474     3107      967    49152        0             0 dietpi-dashboar
Jan 17 00:50:42 Server kernel: [    480]   106   480  1477638   597574  5484544        0             0 java
Jan 17 00:50:42 Server kernel: [    487]     0   487     1017       26    32768        0             0 agetty
Jan 17 00:50:42 Server kernel: [    490]     0   490     1017       27    40960        0             0 agetty
Jan 17 00:50:42 Server kernel: [    636]   101   636    22529      710    73728        0             0 systemd-timesyn
Jan 17 00:50:42 Server kernel: [    698]     0   698    18129      597   122880        0             0 nmbd
Jan 17 00:50:42 Server kernel: [    737]     0   737    22096     1168   155648        0             0 smbd
Jan 17 00:50:42 Server kernel: [    739]     0   739    21615      735   147456        0             0 smbd-notifyd
Jan 17 00:50:42 Server kernel: [    740]     0   740    21617      737   143360        0             0 cleanupd
Jan 17 00:50:42 Server kernel: [    744]   105   744    15628      591   102400        0             0 redis-server
Jan 17 00:50:42 Server kernel: [    825]   104   825   454784    49139   630784        0             0 mariadbd
Jan 17 00:50:42 Server kernel: [    858]     0   858    88563     2119   172032        0             0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [    928]    33   928     2455      661    53248        0             0 lighttpd
Jan 17 00:50:42 Server kernel: [    934]     0   934     1413       63    40960        0             0 cron
Jan 17 00:50:42 Server kernel: [    941]   106   941    25951      139    69632        0             0 controller
Jan 17 00:50:42 Server kernel: [  23222]    33 23222   114889     9336   262144        0             0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [  25388]    33 25388   114397     8882   258048        0             0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [  26025]    33 26025    88563     1467   159744        0             0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [  26242]    33 26242    88563     1467   159744        0             0 php-fpm7.4
Jan 17 00:50:42 Server kernel: [  26272]     0 26272     2228      115    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26273]    33 26273      517       17    32768        0             0 sh
Jan 17 00:50:42 Server kernel: [  26275]    33 26275    63974    10132   233472        0             0 php
Jan 17 00:50:42 Server kernel: [  26292]     0 26292     2136       89    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26297]     0 26297     2136       89    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26298]     0 26298     2136       89    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26301]     0 26301     2136       89    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26304]     0 26304     2136       89    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26308]     0 26308     2136       88    49152        0             0 cron
Jan 17 00:50:42 Server kernel: [  26309]     0 26309    22128      761   139264        0             0 smbd
Jan 17 00:50:42 Server kernel: [  26312]     0 26312     1965       76    45056        0             0 cron
Jan 17 00:50:42 Server kernel: [  26314]     0 26314    18129      597   118784        0             0 nmbd
Jan 17 00:50:42 Server kernel: [  26319]     0 26319     1731       74    45056        0             0 cron
Jan 17 00:50:42 Server kernel: [  26324]     0 26324     1637       71    45056        0             0 cron
Jan 17 00:50:42 Server kernel: [  26326]     0 26326     1461       65    40960        0             0 cron
Jan 17 00:50:42 Server kernel: [  26328]     0 26328     1455       65    40960        0             0 cron
Jan 17 00:50:42 Server kernel: [  26331]     0 26331     1413       63    40960        0             0 cron
Jan 17 00:50:42 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=480,uid=106
Jan 17 00:50:42 Server kernel: Out of memory: Killed process 480 (java) total-vm:5910552kB, anon-rss:2390104kB, file-rss:0kB, shmem-rss:192kB, UID:106 pgtables:5356kB oom_score_adj:0
Jan 17 00:50:42 Server kernel: oom_reaper: reaped process 480 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:32kB
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Failed with result 'signal'.
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Unit process 941 (controller) remains running after unit stopped.
Jan 17 00:50:42 Server systemd[1]: elasticsearch.service: Consumed 17min 11.069s CPU time.
Jan 17 00:50:42 Server systemd[1]: Starting Clean php session files...
Jan 17 00:50:42 Server CRON[26326]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26312]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26328]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26292]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26331]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26298]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26319]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26308]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26304]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26297]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 00:50:42 Server CRON[26301]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26324]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26337]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:50:42 Server CRON[26342]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26345]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26348]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26340]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 00:50:42 Server CRON[26349]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26343]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26341]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26346]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26347]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26350]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 17 00:50:42 Server CRON[26344]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26351]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 17 00:50:42 Server CRON[26339]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:50:42 Server CRON[26326]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:42 Server CRON[26297]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:43 Server CRON[26304]: pam_unix(cron:session): session closed for user root
Jan 17 00:50:44 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 17 00:50:44 Server systemd[1]: Finished Clean php session files.
Jan 17 00:50:47 Server CRON[26308]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26324]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26337]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26331]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26312]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26292]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26319]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26298]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:47 Server CRON[26328]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:50:49 Server CRON[26301]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:53:01 Server CRON[26272]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:55:22 Server CRON[26521]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 00:55:22 Server CRON[26522]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 00:55:23 Server CRON[26521]: pam_unix(cron:session): session closed for user www-data
Jan 17 00:55:59 Server systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
Jan 17 00:56:01 Server fstrim[26530]: /boot: 91.6 MiB (96023040 bytes) trimmed on /dev/mmcblk0p1
Jan 17 00:56:01 Server fstrim[26530]: /: 55.2 GiB (59294412800 bytes) trimmed on /dev/mmcblk0p2
Jan 17 00:56:01 Server systemd[1]: fstrim.service: Succeeded.
Jan 17 00:56:01 Server systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
Jan 17 01:00:06 Server CRON[26603]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:00:06 Server CRON[26604]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:00:07 Server CRON[26603]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:05:07 Server CRON[26609]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:05:07 Server CRON[26610]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:05:23 Server systemd[1]: Starting Certbot...
Jan 17 01:05:25 Server systemd[1]: certbot.service: Succeeded.
Jan 17 01:05:25 Server systemd[1]: Finished Certbot.
Jan 17 01:05:25 Server systemd[1]: certbot.service: Consumed 1.311s CPU time.
Jan 17 01:07:55 Server CRON[26609]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:09:01 Server CRON[26640]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:09:01 Server CRON[26641]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 17 01:09:01 Server CRON[26640]: pam_unix(cron:session): session closed for user root
Jan 17 01:09:01 Server systemd[1]: Starting Clean php session files...
Jan 17 01:09:01 Server systemd[1]: phpsessionclean.service: Succeeded.
Jan 17 01:09:01 Server systemd[1]: Finished Clean php session files.
Jan 17 01:10:19 Server CRON[26690]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:10:19 Server CRON[26691]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:10:21 Server CRON[26690]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:15:12 Server CRON[26770]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:15:12 Server CRON[26771]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:15:13 Server CRON[26770]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:17:12 Server CRON[26776]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:17:12 Server CRON[26777]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 01:17:12 Server CRON[26776]: pam_unix(cron:session): session closed for user root
Jan 17 01:20:12 Server CRON[26862]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:20:12 Server CRON[26863]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:20:14 Server CRON[26862]: pam_unix(cron:session): session closed for user www-data
Jan 17 01:25:12 Server CRON[26871]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 01:25:12 Server CRON[26872]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:12 Server CRON[26873]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Jan 17 01:25:12 Server CRON[26874]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Jan 17 01:25:12 Server systemd[1]: Stopping Network Time Synchronization...
Jan 17 01:25:12 Server systemd[1]: systemd-timesyncd.service: Succeeded.
Jan 17 01:25:12 Server systemd[1]: Stopped Network Time Synchronization.
Jan 17 01:25:12 Server systemd[1]: systemd-timesyncd.service: Consumed 37.432s CPU time.
Jan 17 01:25:13 Server runuser[27028]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:13 Server runuser[27028]: pam_unix(runuser:session): session closed for user www-data
Jan 17 01:25:13 Server runuser[27032]: pam_unix(runuser:session): session opened for user www-data(uid=33) by (uid=0)
Jan 17 01:25:13 Server runuser[27032]: pam_unix(runuser:session): session closed for user www-data
Jan 17 01:25:13 Server CRON[26871]: pam_unix(cron:session): session closed for user root
Jan 17 01:25:13 Server CRON[26872]: pam_unix(cron:session): session closed for user www-data
J
9

This indicates that as well as Elasticsearch you have MariaDB, Redis, and several heavy PHP processes all running on this machine. It just doesn't have enough RAM to cope. The advice from my previous message still stands:

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.