Elasticsearch process ended by code 137

Hi,

When checking the error message for the termination of the Elasticsearch process, it was indicating that the process was terminated due to error 137, when consulting I saw that it indicates excessive memory consumption and subsequent lack of it.

When querying the elasticsearch status, one of the returns is the amount of Xms and Xmx memory allocated (4mb and 64mb), a very low value for the service.

To try to correct the problem, the heap.options file was created within etc/elasticsearch/java.options.d and transferred 5gb to Xms and Xmx, however, whenever I consult the elasticsearch status, the value of 4mb and 64mb is displayed.

When checking the elasticsearch log, the value of 5GB is displayed.

I don't know for sure if it actually accepted the 5GB or if the values ​​are set at 4mb and 64mb.

Can you share the Elasticsearch or system logs for this? How many memory do your system have?

If you set Xms and Xmx to 5GB on a file inside java.options.d, then Elasticsearch will use this as the heap memory.

The process using Xms as 4mb and Xmx as 64mb is not the main Elasticsearch process, not sure exactly what this is but it seens to be related to using cli commands on Elasticsearch, like add/removing things from keystore, maybe someone from Elastic can confirm what is its function.

You can check how many memory the main Elasticsearch process is using looking at the logs or running ps -ef | grep elasticsearch | grep bootstrap

Are you still getting OOM errors?

hi,

here is my system log (jornalctl)

You can check how many memory the main Elasticsearch process is using looking at the logs or running ps -ef | grep elasticsearch | grep bootstrap

about this i have this result, that show me the 5gb

What is the total memory you have in your server? Are you only running Elasticsearch on this server?

Also, since this is a systemd error, could you check the system log for more hints of the issue?

It would be /var/log/messages or /var/log/syslog.

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

1 Like

i have 12GB of RAM and run kibana, elasticsearch and logstash on the server

my environment is small, around 40mb of database growth per day

about the logs

/var/log/messages

Oct 19 03:29:24 elk kernel: systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 19 03:29:24 elk kernel: systemd cpuset=/ mems_allowed=0
Oct 19 03:29:24 elk kernel: CPU: 1 PID: 1 Comm: systemd Tainted: G               ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Oct 19 03:29:24 elk kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/06/2023
Oct 19 03:29:24 elk kernel: Call Trace:
Oct 19 03:29:24 elk kernel: [<ffffffff8ddb1bec>] dump_stack+0x19/0x1f
Oct 19 03:29:24 elk kernel: [<ffffffff8ddacb4f>] dump_header+0x90/0x22d
Oct 19 03:29:24 elk kernel: [<ffffffff8d92229b>] ? cred_has_capability+0x6b/0x130
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cd335>] oom_kill_process+0x2d5/0x4a0
Oct 19 03:29:24 elk kernel: [<ffffffff8d92237c>] ? selinux_capable+0x1c/0x40
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cda3a>] out_of_memory+0x31a/0x500
Oct 19 03:29:24 elk kernel: [<ffffffff8d7d4844>] __alloc_pages_nodemask+0xae4/0xbf0
Oct 19 03:29:24 elk kernel: [<ffffffff8d8259d8>] alloc_pages_current+0x98/0x110
Oct 19 03:29:24 elk kernel: [<ffffffff8d7c8c57>] __page_cache_alloc+0x97/0xb0
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cbce0>] filemap_fault+0x270/0x420
Oct 19 03:29:24 elk kernel: [<ffffffffc027dffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Oct 19 03:29:24 elk kernel: [<ffffffffc027e20c>] xfs_filemap_fault+0x2c/0x40 [xfs]
Oct 19 03:29:24 elk kernel: [<ffffffff8d7fa15a>] __do_fault.isra.61+0x8a/0x100
Oct 19 03:29:24 elk kernel: [<ffffffff8d7fa70c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 19 03:29:24 elk kernel: [<ffffffff8d802113>] handle_mm_fault+0xa33/0x1190
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbf7d3>] __do_page_fault+0x213/0x510
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbfb05>] do_page_fault+0x35/0x90
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbb7b8>] page_fault+0x28/0x30
Oct 19 03:29:24 elk kernel: Mem-Info:
Oct 19 03:29:25 elk kernel: active_anon:2342335 inactive_anon:45955 isolated_anon:0#012 active_file:6684 inactive_file:7648 isolated_file:0#012 unevictable:0 dirty:3 writeback:7 unstable:0#012 slab_reclaimable:9190 slab_unreclaimable:10863#012 mapped:12048 shmem:46023 pagetables:9848 bounce:0#012 free:29957 free_pcp:797 free_cma:0
Oct 19 03:29:25 elk kernel: Node 0 DMA free:15836kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 2809 9815 9815
Oct 19 03:29:25 elk kernel: Node 0 DMA32 free:52664kB min:19308kB low:24132kB high:28960kB active_anon:2715860kB inactive_anon:41932kB active_file:6540kB inactive_file:6772kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3126072kB managed:2879416kB mlocked:0kB dirty:0kB writeback:0kB mapped:4904kB shmem:41956kB slab_reclaimable:6776kB slab_unreclaimable:12348kB kernel_stack:1712kB pagetables:11564kB unstable:0kB bounce:0kB free_pcp:2084kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 7006 7006
Oct 19 03:29:25 elk kernel: Node 0 Normal free:55216kB min:48164kB low:60204kB high:72244kB active_anon:6653480kB inactive_anon:141888kB active_file:21056kB inactive_file:25504kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174352kB mlocked:0kB dirty:12kB writeback:28kB mapped:43288kB shmem:142136kB slab_reclaimable:29984kB slab_unreclaimable:31104kB kernel_stack:5680kB pagetables:27828kB unstable:0kB bounce:0kB free_pcp:2396kB local_pcp:356kB free_cma:0kB writeback_tmp:0kB pages_scanned:50803 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 0 0
Oct 19 03:29:25 elk kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15836kB
Oct 19 03:29:25 elk kernel: Node 0 DMA32: 1313*4kB (UEM) 670*8kB (UEM) 753*16kB (UEM) 456*32kB (UEM) 141*64kB (UEM) 39*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52292kB
Oct 19 03:29:25 elk kernel: Node 0 Normal: 298*4kB (UE) 152*8kB (UE) 1225*16kB (UEM) 601*32kB (UEM) 99*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 19 03:29:25 elk kernel: 63039 total pagecache pages
Oct 19 03:29:25 elk kernel: 0 pages in swap cache
Oct 19 03:29:25 elk kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 19 03:29:25 elk kernel: Free swap  = 0kB
Oct 19 03:29:25 elk kernel: Total swap = 0kB
Oct 19 03:29:25 elk kernel: 2620506 pages RAM
Oct 19 03:29:25 elk kernel: 0 pages HighMem/MovableOnly
Oct 19 03:29:25 elk kernel: 103105 pages reserved
Oct 19 03:29:25 elk kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Oct 19 03:29:25 elk kernel: [  333]     0   333    34457    17617      71        0             0 systemd-journal
Oct 19 03:29:25 elk kernel: [  353]     0   353    48227      754      29        0             0 lvmetad
Oct 19 03:29:25 elk kernel: [  356]     0   356    12157      572      24        0         -1000 systemd-udevd
Oct 19 03:29:25 elk kernel: [  473]     0   473    13883      111      28        0         -1000 auditd
Oct 19 03:29:25 elk kernel: [  502]    81   502    16646      191      32        0          -900 dbus-daemon
Oct 19 03:29:25 elk kernel: [  508]   999   508   153253     2083      63        0             0 polkitd
Oct 19 03:29:25 elk kernel: [  524]     0   524    27552       34       9        0             0 agetty
Oct 19 03:29:25 elk kernel: [  525]     0   525    27552       34      11        0             0 agetty
Oct 19 03:29:25 elk kernel: [  526]     0   526     1097       36       8        0             0 acpid
Oct 19 03:29:25 elk kernel: [  528]   998   528    24067       77      16        0             0 chronyd
Oct 19 03:29:25 elk kernel: [  579]     0   579    89712     5624      93        0             0 firewalld
Oct 19 03:29:25 elk kernel: [  609]     0   609   137038      533      84        0             0 NetworkManager
Oct 19 03:29:25 elk kernel: [  753]     0   753    25751      536      52        0             0 dhclient
Oct 19 03:29:25 elk kernel: [  993]   993   993   225018    13478     125        0             0 apm-server
Oct 19 03:29:25 elk kernel: [ 1003]     0  1003    95070    10297     130        0             0 rsyslogd
Oct 19 03:29:25 elk kernel: [ 1004]     0  1004   143573     3344      96        0             0 tuned
Oct 19 03:29:25 elk kernel: [ 1134]     0  1134    22451      259      44        0             0 master
Oct 19 03:29:25 elk kernel: [ 1138]    89  1138    22494      255      43        0             0 qmgr
Oct 19 03:29:25 elk kernel: [ 3401]     0  3401   181316     2825      23        0          -999 google_guest_ag
Oct 19 03:29:25 elk kernel: [ 3509]     0  3509     6654      152      18        0             0 systemd-logind
Oct 19 03:29:25 elk kernel: [ 3517]     0  3517    31597      158      18        0             0 crond
Oct 19 03:29:25 elk kernel: [ 3871]     0  3871    28250      259      57        0         -1000 sshd
Oct 19 03:29:25 elk kernel: [30128]   994 30128    19753      200      39        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30129]   994 30129    19754      290      39        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30130]   994 30130    19783      327      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30131]   994 30131    19783      326      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30132]   994 30132    19783      327      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30151]     0 30151   300200     6929      80        0             0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30197]     0 30197   282650     4727      72        0             0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30205]     0 30205   345721    14615     130        0             0 otelopscol
Oct 19 03:29:25 elk kernel: [30221]     0 30221   383419     4635     102        0             0 fluent-bit
Oct 19 03:29:25 elk kernel: [ 8141]     0  8141   182383     2933      28        0             0 google_osconfig
Oct 19 03:29:25 elk kernel: [ 8776]   997  8776   680694    20489     101        0             0 java
Oct 19 03:29:25 elk kernel: [ 8843]   997  8843  5848257  1540325    4570        0             0 java
Oct 19 03:29:25 elk kernel: [ 8868]   997  8868    13585      136      19        0             0 controller
Oct 19 03:29:25 elk kernel: [ 9025]   995  9025  1333765   486686    1108        0             0 java
Oct 19 03:29:25 elk kernel: [ 9536]   996  9536  2986432   149879    1717        0             0 node
Oct 19 03:29:25 elk kernel: [ 1039]    89  1039    22477      251      46        0             0 pickup
Oct 19 03:29:25 elk kernel: [ 5196]     0  5196    31372       61      18        0             0 anacron
Oct 19 03:29:25 elk kernel: [ 6607]     0  6607    28321       55      12        0             0 run-parts
Oct 19 03:29:25 elk kernel: [ 6611]     0  6611    87999     3582     124        0             0 yum-cron
Oct 19 03:29:25 elk kernel: [ 6612]     0  6612    28411       38      11        0             0 awk
Oct 19 03:29:25 elk kernel: [ 7200]     0  7200   429944    71913     280        0             0 yum
Oct 19 03:29:25 elk kernel: [ 7240]   994  7240    19783      242      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7241]   994  7241    19783      235      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7245]   994  7245    19783      236      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: Out of memory: Kill process 8843 (java) score 613 or sacrifice child
Oct 19 03:29:25 elk kernel: Killed process 8868 (controller), UID 997, total-vm:54340kB, anon-rss:544kB, file-rss:0kB, shmem-rss:0kB
Oct 19 03:29:25 elk kernel: Ruby-0-Thread-3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 19 03:29:25 elk kernel: Ruby-0-Thread-3 cpuset=/ mems_allowed=0
Oct 19 03:29:25 elk kernel: CPU: 0 PID: 9193 Comm: Ruby-0-Thread-3 Tainted: G               ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Oct 19 03:29:25 elk kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/06/2023
Oct 19 03:29:25 elk kernel: Call Trace:
Oct 19 03:29:25 elk kernel: [<ffffffff8ddb1bec>] dump_stack+0x19/0x1f
Oct 19 03:29:25 elk kernel: [<ffffffff8ddacb4f>] dump_header+0x90/0x22d
Oct 19 03:29:25 elk kernel: [<ffffffff8d92229b>] ? cred_has_capability+0x6b/0x130
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cd335>] oom_kill_process+0x2d5/0x4a0
Oct 19 03:29:25 elk kernel: [<ffffffff8d92237c>] ? selinux_capable+0x1c/0x40
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cda3a>] out_of_memory+0x31a/0x500
Oct 19 03:29:25 elk kernel: [<ffffffff8d7d4844>] __alloc_pages_nodemask+0xae4/0xbf0
Oct 19 03:29:25 elk kernel: [<ffffffff8d8259d8>] alloc_pages_current+0x98/0x110
Oct 19 03:29:25 elk kernel: [<ffffffff8d7c8c57>] __page_cache_alloc+0x97/0xb0
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cbce0>] filemap_fault+0x270/0x420
Oct 19 03:29:25 elk kernel: [<ffffffffc027dffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Oct 19 03:29:25 elk kernel: [<ffffffffc027e20c>] xfs_filemap_fault+0x2c/0x40 [xfs]
Oct 19 03:29:25 elk kernel: [<ffffffff8d7fa15a>] __do_fault.isra.61+0x8a/0x100
Oct 19 03:29:25 elk kernel: [<ffffffff8d7fa70c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 19 03:29:25 elk kernel: [<ffffffff8d802113>] handle_mm_fault+0xa33/0x1190
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbf7d3>] __do_page_fault+0x213/0x510
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbfb05>] do_page_fault+0x35/0x90
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbb7b8>] page_fault+0x28/0x30
Oct 19 03:29:25 elk kernel: Mem-Info:
Oct 19 03:29:25 elk kernel: active_anon:2342164 inactive_anon:45955 isolated_anon:0#012 active_file:4999 inactive_file:5404 isolated_file:0#012 unevictable:0 dirty:2 writeback:4 unstable:0#012 slab_reclaimable:9186 slab_unreclaimable:10862#012 mapped:12102 shmem:46023 pagetables:9829 bounce:0#012 free:30146 free_pcp:874 free_cma:0
Oct 19 03:29:25 elk kernel: Node 0 DMA free:15836kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 2809 9815 9815
Oct 19 03:29:25 elk kernel: Node 0 DMA32 free:54120kB min:19308kB low:24132kB high:28960kB active_anon:2715516kB inactive_anon:41932kB active_file:4320kB inactive_file:7020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3126072kB managed:2879416kB mlocked:0kB dirty:0kB writeback:0kB mapped:5128kB shmem:41956kB slab_reclaimable:6760kB slab_unreclaimable:12348kB kernel_stack:1696kB pagetables:11552kB unstable:0kB bounce:0kB free_pcp:1516kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 7006 7006
Oct 19 03:29:25 elk kernel: Node 0 Normal free:60784kB min:48164kB low:60204kB high:72244kB active_anon:6653140kB inactive_anon:141888kB active_file:18356kB inactive_file:22252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174352kB mlocked:0kB dirty:8kB writeback:16kB mapped:43280kB shmem:142136kB slab_reclaimable:29984kB slab_unreclaimable:31100kB kernel_stack:5616kB pagetables:27764kB unstable:0kB bounce:0kB free_pcp:1548kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 0 0
Oct 19 03:29:25 elk kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15836kB
Oct 19 03:29:25 elk kernel: Node 0 DMA32: 1530*4kB (UEM) 978*8kB (UEM) 724*16kB (UEM) 446*32kB (UEM) 142*64kB (UEM) 35*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54392kB
Oct 19 03:29:25 elk kernel: Node 0 Normal: 1902*4kB (UEM) 765*8kB (UEM) 1439*16kB (UEM) 650*32kB (UEM) 105*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64272kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 19 03:29:25 elk kernel: 58898 total pagecache pages
Oct 19 03:29:25 elk kernel: 0 pages in swap cache
Oct 19 03:29:25 elk kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 19 03:29:25 elk kernel: Free swap  = 0kB
Oct 19 03:29:25 elk kernel: Total swap = 0kB
Oct 19 03:29:25 elk kernel: 2620506 pages RAM
Oct 19 03:29:25 elk kernel: 0 pages HighMem/MovableOnly
Oct 19 03:29:25 elk kernel: 103105 pages reserved
Oct 19 03:29:25 elk kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Oct 19 03:29:25 elk kernel: [  333]     0   333    34457    17637      71        0             0 systemd-journal
Oct 19 03:29:25 elk kernel: [  353]     0   353    48227      754      29        0             0 lvmetad
Oct 19 03:29:25 elk kernel: [  356]     0   356    12157      572      24        0         -1000 systemd-udevd
Oct 19 03:29:25 elk kernel: [  473]     0   473    13883      111      28        0         -1000 auditd
Oct 19 03:29:25 elk kernel: [  502]    81   502    16646      191      32        0          -900 dbus-daemon
Oct 19 03:29:25 elk kernel: [  508]   999   508   153253     2083      63        0             0 polkitd
Oct 19 03:29:25 elk kernel: [  524]     0   524    27552       34       9        0             0 agetty
Oct 19 03:29:25 elk kernel: [  525]     0   525    27552       34      11        0             0 agetty
Oct 19 03:29:25 elk kernel: [  526]     0   526     1097       36       8        0             0 acpid
Oct 19 03:29:25 elk kernel: [  528]   998   528    24067       77      16        0             0 chronyd
Oct 19 03:29:25 elk kernel: [  579]     0   579    89712     5624      93        0             0 firewalld
Oct 19 03:29:25 elk kernel: [  609]     0   609   137038      533      84        0             0 NetworkManager
Oct 19 03:29:25 elk kernel: [  753]     0   753    25751      536      52        0             0 dhclient
Oct 19 03:29:25 elk kernel: [  993]   993   993   225018    13478     125        0             0 apm-server
Oct 19 03:29:25 elk kernel: [ 1003]     0  1003    95070    10297     130        0             0 rsyslogd
Oct 19 03:29:25 elk kernel: [ 1004]     0  1004   143573     3344      96        0             0 tuned
Oct 19 03:29:25 elk kernel: [ 1134]     0  1134    22451      266      44        0             0 master
Oct 19 03:29:25 elk kernel: [ 1138]    89  1138    22494      255      43        0             0 qmgr
Oct 19 03:29:25 elk kernel: [ 3401]     0  3401   181316     2825      23        0          -999 google_guest_ag
Oct 19 03:29:25 elk kernel: [ 3509]     0  3509     6654      152      18        0             0 systemd-logind
Oct 19 03:29:25 elk kernel: [ 3517]     0  3517    31597      158      18        0             0 crond
Oct 19 03:29:25 elk kernel: [ 3871]     0  3871    28250      259      57        0         -1000 sshd
Oct 19 03:29:25 elk kernel: [30128]   994 30128    19753      200      39        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30129]   994 30129    19753      304      39        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30130]   994 30130    19783      327      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30131]   994 30131    19783      326      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30132]   994 30132    19783      327      40        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30151]     0 30151   300200     6929      80        0             0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30197]     0 30197   282650     4727      72        0             0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30205]     0 30205   345721    14615     130        0             0 otelopscol
Oct 19 03:29:25 elk kernel: [30221]     0 30221   383419     4635     102        0             0 fluent-bit
Oct 19 03:29:25 elk kernel: [ 8141]     0  8141   182383     2933      28        0             0 google_osconfig
Oct 19 03:29:25 elk kernel: [ 8776]   997  8776   680694    20489     101        0             0 java
Oct 19 03:29:25 elk kernel: [ 8843]   997  8843  5848258  1540305    4570        0             0 java
Oct 19 03:29:25 elk kernel: [ 9025]   995  9025  1333764   486688    1108        0             0 java
Oct 19 03:29:25 elk kernel: [ 9536]   996  9536  2986432   149879    1717        0             0 node
Oct 19 03:29:25 elk kernel: [ 1039]    89  1039    22477      251      46        0             0 pickup
Oct 19 03:29:25 elk kernel: [ 5196]     0  5196    31372       61      18        0             0 anacron
Oct 19 03:29:25 elk kernel: [ 6607]     0  6607    28321       55      12        0             0 run-parts
Oct 19 03:29:25 elk kernel: [ 6611]     0  6611    87999     3582     124        0             0 yum-cron
Oct 19 03:29:25 elk kernel: [ 6612]     0  6612    28411       38      11        0             0 awk
Oct 19 03:29:25 elk kernel: [ 7200]     0  7200   429944    71917     280        0             0 yum
Oct 19 03:29:25 elk kernel: [ 7240]   994  7240    19783      236      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7241]   994  7241    19783      235      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7245]   994  7245    19783      236      42        0             0 zabbix_agentd
Oct 19 03:29:25 elk kernel: Out of memory: Kill process 8843 (java) score 613 or sacrifice child
Oct 19 03:29:25 elk kernel: Killed process 8843 (java), UID 997, total-vm:23393032kB, anon-rss:6161220kB, file-rss:0kB, shmem-rss:0kB
Oct 19 03:29:25 elk logstash: [2023-10-19T03:29:25,090][WARN ][logstash.inputs.jdbc     ][main][68a1449437af78762b8d8badb96e84487ae331fb31ed94033ab426ab38f5c6f2] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLException: [Cache JDBC] Communication link failure: Connection timed out", :cause=>"#<Java::JavaSql::SQLException: [Cache JDBC] Communication link failure: Connection timed out>"}
Oct 19 03:29:25 elk logstash: [2023-10-19T03:29:25,631][ERROR][logstash.inputs.jdbc     ][main][68a1449437af78762b8d8badb96e84487ae331fb31ed94033ab426ab38f5c6f2] Unable to execute statement. Tried 1 times.
Oct 19 03:29:26 elk kibana: [2023-10-19T03:29:26.631-03:00][INFO ][status] Kibana is now unavailable (was available)
Oct 19 03:29:27 elk systemd-entrypoint: ERROR: Elasticsearch exited unexpectedly, with exit code 137

Yeah, having the entire stack on the same machine can give you some issues.

Both Logstash and Elasticsearch can be memory intensive, do you have anything memory intensive running on this same server?

How much memory did you configure for Logstash? It is the jvm.options file inside /etc/logstash.

No, this server is only for ELK Stack

to logstash i have 1gb to Xms and Xmx

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.