Hi,
When checking the error message for the termination of the Elasticsearch process, it was indicating that the process was terminated due to error 137, when consulting I saw that it indicates excessive memory consumption and subsequent lack of it.
When querying the elasticsearch status, one of the returns is the amount of Xms and Xmx memory allocated (4mb and 64mb), a very low value for the service.
To try to correct the problem, the heap.options file was created within etc/elasticsearch/java.options.d and transferred 5gb to Xms and Xmx, however, whenever I consult the elasticsearch status, the value of 4mb and 64mb is displayed.
When checking the elasticsearch log, the value of 5GB is displayed.
I don't know for sure if it actually accepted the 5GB or if the values are set at 4mb and 64mb.
leandrojmp
October 19, 2023, 3:50pm
2
Can you share the Elasticsearch or system logs for this? How many memory do your system have?
If you set Xms and Xmx to 5GB on a file inside java.options.d, then Elasticsearch will use this as the heap memory.
The process using Xms as 4mb and Xmx as 64mb is not the main Elasticsearch process, not sure exactly what this is but it seens to be related to using cli commands on Elasticsearch, like add/removing things from keystore, maybe someone from Elastic can confirm what is its function.
You can check how many memory the main Elasticsearch process is using looking at the logs or running ps -ef | grep elasticsearch | grep bootstrap
Are you still getting OOM errors?
hi,
here is my system log (jornalctl)
You can check how many memory the main Elasticsearch process is using looking at the logs or running ps -ef | grep elasticsearch | grep bootstrap
about this i have this result, that show me the 5gb
leandrojmp
October 19, 2023, 5:16pm
4
What is the total memory you have in your server? Are you only running Elasticsearch on this server?
Also, since this is a systemd error, could you check the system log for more hints of the issue?
It would be /var/log/messages or /var/log/syslog.
dadoonet
October 19, 2023, 5:31pm
5
Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.
Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
2 Likes
i have 12GB of RAM and run kibana, elasticsearch and logstash on the server
my environment is small, around 40mb of database growth per day
about the logs
/var/log/messages
Oct 19 03:29:24 elk kernel: systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 19 03:29:24 elk kernel: systemd cpuset=/ mems_allowed=0
Oct 19 03:29:24 elk kernel: CPU: 1 PID: 1 Comm: systemd Tainted: G ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Oct 19 03:29:24 elk kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/06/2023
Oct 19 03:29:24 elk kernel: Call Trace:
Oct 19 03:29:24 elk kernel: [<ffffffff8ddb1bec>] dump_stack+0x19/0x1f
Oct 19 03:29:24 elk kernel: [<ffffffff8ddacb4f>] dump_header+0x90/0x22d
Oct 19 03:29:24 elk kernel: [<ffffffff8d92229b>] ? cred_has_capability+0x6b/0x130
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cd335>] oom_kill_process+0x2d5/0x4a0
Oct 19 03:29:24 elk kernel: [<ffffffff8d92237c>] ? selinux_capable+0x1c/0x40
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cda3a>] out_of_memory+0x31a/0x500
Oct 19 03:29:24 elk kernel: [<ffffffff8d7d4844>] __alloc_pages_nodemask+0xae4/0xbf0
Oct 19 03:29:24 elk kernel: [<ffffffff8d8259d8>] alloc_pages_current+0x98/0x110
Oct 19 03:29:24 elk kernel: [<ffffffff8d7c8c57>] __page_cache_alloc+0x97/0xb0
Oct 19 03:29:24 elk kernel: [<ffffffff8d7cbce0>] filemap_fault+0x270/0x420
Oct 19 03:29:24 elk kernel: [<ffffffffc027dffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Oct 19 03:29:24 elk kernel: [<ffffffffc027e20c>] xfs_filemap_fault+0x2c/0x40 [xfs]
Oct 19 03:29:24 elk kernel: [<ffffffff8d7fa15a>] __do_fault.isra.61+0x8a/0x100
Oct 19 03:29:24 elk kernel: [<ffffffff8d7fa70c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 19 03:29:24 elk kernel: [<ffffffff8d802113>] handle_mm_fault+0xa33/0x1190
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbf7d3>] __do_page_fault+0x213/0x510
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbfb05>] do_page_fault+0x35/0x90
Oct 19 03:29:24 elk kernel: [<ffffffff8ddbb7b8>] page_fault+0x28/0x30
Oct 19 03:29:24 elk kernel: Mem-Info:
Oct 19 03:29:25 elk kernel: active_anon:2342335 inactive_anon:45955 isolated_anon:0#012 active_file:6684 inactive_file:7648 isolated_file:0#012 unevictable:0 dirty:3 writeback:7 unstable:0#012 slab_reclaimable:9190 slab_unreclaimable:10863#012 mapped:12048 shmem:46023 pagetables:9848 bounce:0#012 free:29957 free_pcp:797 free_cma:0
Oct 19 03:29:25 elk kernel: Node 0 DMA free:15836kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 2809 9815 9815
Oct 19 03:29:25 elk kernel: Node 0 DMA32 free:52664kB min:19308kB low:24132kB high:28960kB active_anon:2715860kB inactive_anon:41932kB active_file:6540kB inactive_file:6772kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3126072kB managed:2879416kB mlocked:0kB dirty:0kB writeback:0kB mapped:4904kB shmem:41956kB slab_reclaimable:6776kB slab_unreclaimable:12348kB kernel_stack:1712kB pagetables:11564kB unstable:0kB bounce:0kB free_pcp:2084kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 7006 7006
Oct 19 03:29:25 elk kernel: Node 0 Normal free:55216kB min:48164kB low:60204kB high:72244kB active_anon:6653480kB inactive_anon:141888kB active_file:21056kB inactive_file:25504kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174352kB mlocked:0kB dirty:12kB writeback:28kB mapped:43288kB shmem:142136kB slab_reclaimable:29984kB slab_unreclaimable:31104kB kernel_stack:5680kB pagetables:27828kB unstable:0kB bounce:0kB free_pcp:2396kB local_pcp:356kB free_cma:0kB writeback_tmp:0kB pages_scanned:50803 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 0 0
Oct 19 03:29:25 elk kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15836kB
Oct 19 03:29:25 elk kernel: Node 0 DMA32: 1313*4kB (UEM) 670*8kB (UEM) 753*16kB (UEM) 456*32kB (UEM) 141*64kB (UEM) 39*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52292kB
Oct 19 03:29:25 elk kernel: Node 0 Normal: 298*4kB (UE) 152*8kB (UE) 1225*16kB (UEM) 601*32kB (UEM) 99*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 19 03:29:25 elk kernel: 63039 total pagecache pages
Oct 19 03:29:25 elk kernel: 0 pages in swap cache
Oct 19 03:29:25 elk kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 19 03:29:25 elk kernel: Free swap = 0kB
Oct 19 03:29:25 elk kernel: Total swap = 0kB
Oct 19 03:29:25 elk kernel: 2620506 pages RAM
Oct 19 03:29:25 elk kernel: 0 pages HighMem/MovableOnly
Oct 19 03:29:25 elk kernel: 103105 pages reserved
Oct 19 03:29:25 elk kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 19 03:29:25 elk kernel: [ 333] 0 333 34457 17617 71 0 0 systemd-journal
Oct 19 03:29:25 elk kernel: [ 353] 0 353 48227 754 29 0 0 lvmetad
Oct 19 03:29:25 elk kernel: [ 356] 0 356 12157 572 24 0 -1000 systemd-udevd
Oct 19 03:29:25 elk kernel: [ 473] 0 473 13883 111 28 0 -1000 auditd
Oct 19 03:29:25 elk kernel: [ 502] 81 502 16646 191 32 0 -900 dbus-daemon
Oct 19 03:29:25 elk kernel: [ 508] 999 508 153253 2083 63 0 0 polkitd
Oct 19 03:29:25 elk kernel: [ 524] 0 524 27552 34 9 0 0 agetty
Oct 19 03:29:25 elk kernel: [ 525] 0 525 27552 34 11 0 0 agetty
Oct 19 03:29:25 elk kernel: [ 526] 0 526 1097 36 8 0 0 acpid
Oct 19 03:29:25 elk kernel: [ 528] 998 528 24067 77 16 0 0 chronyd
Oct 19 03:29:25 elk kernel: [ 579] 0 579 89712 5624 93 0 0 firewalld
Oct 19 03:29:25 elk kernel: [ 609] 0 609 137038 533 84 0 0 NetworkManager
Oct 19 03:29:25 elk kernel: [ 753] 0 753 25751 536 52 0 0 dhclient
Oct 19 03:29:25 elk kernel: [ 993] 993 993 225018 13478 125 0 0 apm-server
Oct 19 03:29:25 elk kernel: [ 1003] 0 1003 95070 10297 130 0 0 rsyslogd
Oct 19 03:29:25 elk kernel: [ 1004] 0 1004 143573 3344 96 0 0 tuned
Oct 19 03:29:25 elk kernel: [ 1134] 0 1134 22451 259 44 0 0 master
Oct 19 03:29:25 elk kernel: [ 1138] 89 1138 22494 255 43 0 0 qmgr
Oct 19 03:29:25 elk kernel: [ 3401] 0 3401 181316 2825 23 0 -999 google_guest_ag
Oct 19 03:29:25 elk kernel: [ 3509] 0 3509 6654 152 18 0 0 systemd-logind
Oct 19 03:29:25 elk kernel: [ 3517] 0 3517 31597 158 18 0 0 crond
Oct 19 03:29:25 elk kernel: [ 3871] 0 3871 28250 259 57 0 -1000 sshd
Oct 19 03:29:25 elk kernel: [30128] 994 30128 19753 200 39 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30129] 994 30129 19754 290 39 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30130] 994 30130 19783 327 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30131] 994 30131 19783 326 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30132] 994 30132 19783 327 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30151] 0 30151 300200 6929 80 0 0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30197] 0 30197 282650 4727 72 0 0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30205] 0 30205 345721 14615 130 0 0 otelopscol
Oct 19 03:29:25 elk kernel: [30221] 0 30221 383419 4635 102 0 0 fluent-bit
Oct 19 03:29:25 elk kernel: [ 8141] 0 8141 182383 2933 28 0 0 google_osconfig
Oct 19 03:29:25 elk kernel: [ 8776] 997 8776 680694 20489 101 0 0 java
Oct 19 03:29:25 elk kernel: [ 8843] 997 8843 5848257 1540325 4570 0 0 java
Oct 19 03:29:25 elk kernel: [ 8868] 997 8868 13585 136 19 0 0 controller
Oct 19 03:29:25 elk kernel: [ 9025] 995 9025 1333765 486686 1108 0 0 java
Oct 19 03:29:25 elk kernel: [ 9536] 996 9536 2986432 149879 1717 0 0 node
Oct 19 03:29:25 elk kernel: [ 1039] 89 1039 22477 251 46 0 0 pickup
Oct 19 03:29:25 elk kernel: [ 5196] 0 5196 31372 61 18 0 0 anacron
Oct 19 03:29:25 elk kernel: [ 6607] 0 6607 28321 55 12 0 0 run-parts
Oct 19 03:29:25 elk kernel: [ 6611] 0 6611 87999 3582 124 0 0 yum-cron
Oct 19 03:29:25 elk kernel: [ 6612] 0 6612 28411 38 11 0 0 awk
Oct 19 03:29:25 elk kernel: [ 7200] 0 7200 429944 71913 280 0 0 yum
Oct 19 03:29:25 elk kernel: [ 7240] 994 7240 19783 242 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7241] 994 7241 19783 235 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7245] 994 7245 19783 236 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: Out of memory: Kill process 8843 (java) score 613 or sacrifice child
Oct 19 03:29:25 elk kernel: Killed process 8868 (controller), UID 997, total-vm:54340kB, anon-rss:544kB, file-rss:0kB, shmem-rss:0kB
Oct 19 03:29:25 elk kernel: Ruby-0-Thread-3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 19 03:29:25 elk kernel: Ruby-0-Thread-3 cpuset=/ mems_allowed=0
Oct 19 03:29:25 elk kernel: CPU: 0 PID: 9193 Comm: Ruby-0-Thread-3 Tainted: G ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Oct 19 03:29:25 elk kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/06/2023
Oct 19 03:29:25 elk kernel: Call Trace:
Oct 19 03:29:25 elk kernel: [<ffffffff8ddb1bec>] dump_stack+0x19/0x1f
Oct 19 03:29:25 elk kernel: [<ffffffff8ddacb4f>] dump_header+0x90/0x22d
Oct 19 03:29:25 elk kernel: [<ffffffff8d92229b>] ? cred_has_capability+0x6b/0x130
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cd335>] oom_kill_process+0x2d5/0x4a0
Oct 19 03:29:25 elk kernel: [<ffffffff8d92237c>] ? selinux_capable+0x1c/0x40
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cda3a>] out_of_memory+0x31a/0x500
Oct 19 03:29:25 elk kernel: [<ffffffff8d7d4844>] __alloc_pages_nodemask+0xae4/0xbf0
Oct 19 03:29:25 elk kernel: [<ffffffff8d8259d8>] alloc_pages_current+0x98/0x110
Oct 19 03:29:25 elk kernel: [<ffffffff8d7c8c57>] __page_cache_alloc+0x97/0xb0
Oct 19 03:29:25 elk kernel: [<ffffffff8d7cbce0>] filemap_fault+0x270/0x420
Oct 19 03:29:25 elk kernel: [<ffffffffc027dffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Oct 19 03:29:25 elk kernel: [<ffffffffc027e20c>] xfs_filemap_fault+0x2c/0x40 [xfs]
Oct 19 03:29:25 elk kernel: [<ffffffff8d7fa15a>] __do_fault.isra.61+0x8a/0x100
Oct 19 03:29:25 elk kernel: [<ffffffff8d7fa70c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 19 03:29:25 elk kernel: [<ffffffff8d802113>] handle_mm_fault+0xa33/0x1190
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbf7d3>] __do_page_fault+0x213/0x510
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbfb05>] do_page_fault+0x35/0x90
Oct 19 03:29:25 elk kernel: [<ffffffff8ddbb7b8>] page_fault+0x28/0x30
Oct 19 03:29:25 elk kernel: Mem-Info:
Oct 19 03:29:25 elk kernel: active_anon:2342164 inactive_anon:45955 isolated_anon:0#012 active_file:4999 inactive_file:5404 isolated_file:0#012 unevictable:0 dirty:2 writeback:4 unstable:0#012 slab_reclaimable:9186 slab_unreclaimable:10862#012 mapped:12102 shmem:46023 pagetables:9829 bounce:0#012 free:30146 free_pcp:874 free_cma:0
Oct 19 03:29:25 elk kernel: Node 0 DMA free:15836kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 2809 9815 9815
Oct 19 03:29:25 elk kernel: Node 0 DMA32 free:54120kB min:19308kB low:24132kB high:28960kB active_anon:2715516kB inactive_anon:41932kB active_file:4320kB inactive_file:7020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3126072kB managed:2879416kB mlocked:0kB dirty:0kB writeback:0kB mapped:5128kB shmem:41956kB slab_reclaimable:6760kB slab_unreclaimable:12348kB kernel_stack:1696kB pagetables:11552kB unstable:0kB bounce:0kB free_pcp:1516kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 7006 7006
Oct 19 03:29:25 elk kernel: Node 0 Normal free:60784kB min:48164kB low:60204kB high:72244kB active_anon:6653140kB inactive_anon:141888kB active_file:18356kB inactive_file:22252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174352kB mlocked:0kB dirty:8kB writeback:16kB mapped:43280kB shmem:142136kB slab_reclaimable:29984kB slab_unreclaimable:31100kB kernel_stack:5616kB pagetables:27764kB unstable:0kB bounce:0kB free_pcp:1548kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 19 03:29:25 elk kernel: lowmem_reserve[]: 0 0 0 0
Oct 19 03:29:25 elk kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15836kB
Oct 19 03:29:25 elk kernel: Node 0 DMA32: 1530*4kB (UEM) 978*8kB (UEM) 724*16kB (UEM) 446*32kB (UEM) 142*64kB (UEM) 35*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54392kB
Oct 19 03:29:25 elk kernel: Node 0 Normal: 1902*4kB (UEM) 765*8kB (UEM) 1439*16kB (UEM) 650*32kB (UEM) 105*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64272kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 19 03:29:25 elk kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 19 03:29:25 elk kernel: 58898 total pagecache pages
Oct 19 03:29:25 elk kernel: 0 pages in swap cache
Oct 19 03:29:25 elk kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 19 03:29:25 elk kernel: Free swap = 0kB
Oct 19 03:29:25 elk kernel: Total swap = 0kB
Oct 19 03:29:25 elk kernel: 2620506 pages RAM
Oct 19 03:29:25 elk kernel: 0 pages HighMem/MovableOnly
Oct 19 03:29:25 elk kernel: 103105 pages reserved
Oct 19 03:29:25 elk kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 19 03:29:25 elk kernel: [ 333] 0 333 34457 17637 71 0 0 systemd-journal
Oct 19 03:29:25 elk kernel: [ 353] 0 353 48227 754 29 0 0 lvmetad
Oct 19 03:29:25 elk kernel: [ 356] 0 356 12157 572 24 0 -1000 systemd-udevd
Oct 19 03:29:25 elk kernel: [ 473] 0 473 13883 111 28 0 -1000 auditd
Oct 19 03:29:25 elk kernel: [ 502] 81 502 16646 191 32 0 -900 dbus-daemon
Oct 19 03:29:25 elk kernel: [ 508] 999 508 153253 2083 63 0 0 polkitd
Oct 19 03:29:25 elk kernel: [ 524] 0 524 27552 34 9 0 0 agetty
Oct 19 03:29:25 elk kernel: [ 525] 0 525 27552 34 11 0 0 agetty
Oct 19 03:29:25 elk kernel: [ 526] 0 526 1097 36 8 0 0 acpid
Oct 19 03:29:25 elk kernel: [ 528] 998 528 24067 77 16 0 0 chronyd
Oct 19 03:29:25 elk kernel: [ 579] 0 579 89712 5624 93 0 0 firewalld
Oct 19 03:29:25 elk kernel: [ 609] 0 609 137038 533 84 0 0 NetworkManager
Oct 19 03:29:25 elk kernel: [ 753] 0 753 25751 536 52 0 0 dhclient
Oct 19 03:29:25 elk kernel: [ 993] 993 993 225018 13478 125 0 0 apm-server
Oct 19 03:29:25 elk kernel: [ 1003] 0 1003 95070 10297 130 0 0 rsyslogd
Oct 19 03:29:25 elk kernel: [ 1004] 0 1004 143573 3344 96 0 0 tuned
Oct 19 03:29:25 elk kernel: [ 1134] 0 1134 22451 266 44 0 0 master
Oct 19 03:29:25 elk kernel: [ 1138] 89 1138 22494 255 43 0 0 qmgr
Oct 19 03:29:25 elk kernel: [ 3401] 0 3401 181316 2825 23 0 -999 google_guest_ag
Oct 19 03:29:25 elk kernel: [ 3509] 0 3509 6654 152 18 0 0 systemd-logind
Oct 19 03:29:25 elk kernel: [ 3517] 0 3517 31597 158 18 0 0 crond
Oct 19 03:29:25 elk kernel: [ 3871] 0 3871 28250 259 57 0 -1000 sshd
Oct 19 03:29:25 elk kernel: [30128] 994 30128 19753 200 39 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30129] 994 30129 19753 304 39 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30130] 994 30130 19783 327 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30131] 994 30131 19783 326 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30132] 994 30132 19783 327 40 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [30151] 0 30151 300200 6929 80 0 0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30197] 0 30197 282650 4727 72 0 0 google_cloud_op
Oct 19 03:29:25 elk kernel: [30205] 0 30205 345721 14615 130 0 0 otelopscol
Oct 19 03:29:25 elk kernel: [30221] 0 30221 383419 4635 102 0 0 fluent-bit
Oct 19 03:29:25 elk kernel: [ 8141] 0 8141 182383 2933 28 0 0 google_osconfig
Oct 19 03:29:25 elk kernel: [ 8776] 997 8776 680694 20489 101 0 0 java
Oct 19 03:29:25 elk kernel: [ 8843] 997 8843 5848258 1540305 4570 0 0 java
Oct 19 03:29:25 elk kernel: [ 9025] 995 9025 1333764 486688 1108 0 0 java
Oct 19 03:29:25 elk kernel: [ 9536] 996 9536 2986432 149879 1717 0 0 node
Oct 19 03:29:25 elk kernel: [ 1039] 89 1039 22477 251 46 0 0 pickup
Oct 19 03:29:25 elk kernel: [ 5196] 0 5196 31372 61 18 0 0 anacron
Oct 19 03:29:25 elk kernel: [ 6607] 0 6607 28321 55 12 0 0 run-parts
Oct 19 03:29:25 elk kernel: [ 6611] 0 6611 87999 3582 124 0 0 yum-cron
Oct 19 03:29:25 elk kernel: [ 6612] 0 6612 28411 38 11 0 0 awk
Oct 19 03:29:25 elk kernel: [ 7200] 0 7200 429944 71917 280 0 0 yum
Oct 19 03:29:25 elk kernel: [ 7240] 994 7240 19783 236 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7241] 994 7241 19783 235 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: [ 7245] 994 7245 19783 236 42 0 0 zabbix_agentd
Oct 19 03:29:25 elk kernel: Out of memory: Kill process 8843 (java) score 613 or sacrifice child
Oct 19 03:29:25 elk kernel: Killed process 8843 (java), UID 997, total-vm:23393032kB, anon-rss:6161220kB, file-rss:0kB, shmem-rss:0kB
Oct 19 03:29:25 elk logstash: [2023-10-19T03:29:25,090][WARN ][logstash.inputs.jdbc ][main][68a1449437af78762b8d8badb96e84487ae331fb31ed94033ab426ab38f5c6f2] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLException: [Cache JDBC] Communication link failure: Connection timed out", :cause=>"#<Java::JavaSql::SQLException: [Cache JDBC] Communication link failure: Connection timed out>"}
Oct 19 03:29:25 elk logstash: [2023-10-19T03:29:25,631][ERROR][logstash.inputs.jdbc ][main][68a1449437af78762b8d8badb96e84487ae331fb31ed94033ab426ab38f5c6f2] Unable to execute statement. Tried 1 times.
Oct 19 03:29:26 elk kibana: [2023-10-19T03:29:26.631-03:00][INFO ][status] Kibana is now unavailable (was available)
Oct 19 03:29:27 elk systemd-entrypoint: ERROR: Elasticsearch exited unexpectedly, with exit code 137
leandrojmp
October 19, 2023, 6:19pm
7
Yeah, having the entire stack on the same machine can give you some issues.
Both Logstash and Elasticsearch can be memory intensive, do you have anything memory intensive running on this same server?
How much memory did you configure for Logstash? It is the jvm.options file inside /etc/logstash.
No, this server is only for ELK Stack
to logstash i have 1gb to Xms and Xmx
system
November 16, 2023, 7:10pm
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
