Can somebody tell me what is in this log?

I can not start Elastic although I can do it few days before.

root@vmi503579:~# journalctl -ex
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 884825]    33 884825    59113     6477   241664      686             0 apache2
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885861]     0 885861     2771      158    57344       11             0 cron
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885862]     0 885862      652       17    40960        0             0 sh
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885863]     0 885863     2373       51    57344        0             0 bash
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885867]     0 885867     2969       70    65536        0             0 systemctl
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885868]   111 885868  3134350  2181589 17993728        0             0 java
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886065]   111 886065    27101      148    86016        0             0 controller
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886080]     0 886080     2771      159    57344       11             0 cron
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886081]     0 886081      652       16    45056        0             0 sh
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886082]     0 886082     2373       60    53248        0             0 bash
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886086]     0 886086     2969       68    69632        0             0 systemctl
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=885868,uid=111
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: Out of memory: Killed process 885868 (java) total-vm:12537400kB, anon-rss:8726356kB, file-rss:0kB, shmem-rss:0kB, UID:111 pgtables:17572kB oom_score_adj:0
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=193.27.228.58 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=20932 PROTO=TCP SPT=50>
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom_reaper: reaped process 885868 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit elasticsearch.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 9.
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit elasticsearch.service has entered the 'failed' state with result 'signal'.
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has finished with a failure.
--
-- The job identifier is 419853 and the job result is failed.
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: (CRON) info (No MTA installed, discarding output)
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: (CRON) info (No MTA installed, discarding output)
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: pam_unix(cron:session): session closed for user root
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: pam_unix(cron:session): session closed for user root
Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886113]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886114]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh)
Jul 13 15:28:01 vmi503579.contaboserver.net systemd[1]: Starting Elasticsearch...
-- Subject: A start job for unit elasticsearch.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has begun execution.
--
-- The job identifier is 419985.
Jul 13 15:28:08 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=26420 PROTO=TCP SPT=5>
Jul 13 15:28:30 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=60186 PROTO=TCP SPT=5>
Jul 13 15:28:53 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=47.100.237.187 DST=173.249.58.30 LEN=40 TOS=0x14 PREC=0x00 TTL=241 ID=33623 PROTO=TCP SPT=4>
Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886403]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886404]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh)
Jul 13 15:29:21 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=185.53.90.85 DST=173.249.58.30 LEN=58 TOS=0x08 PREC=0x00 TTL=246 ID=54321 PROTO=UDP SPT=448>
lines 1157-1213/1213 (END)
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 884825]    33 884825    59113     6477   241664      686             0 apache2
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885861]     0 885861     2771      158    57344       11             0 cron
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885862]     0 885862      652       17    40960        0             0 sh
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885863]     0 885863     2373       51    57344        0             0 bash
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885867]     0 885867     2969       70    65536        0             0 systemctl
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 885868]   111 885868  3134350  2181589 17993728        0             0 java
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886065]   111 886065    27101      148    86016        0             0 controller
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886080]     0 886080     2771      159    57344       11             0 cron
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886081]     0 886081      652       16    45056        0             0 sh
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886082]     0 886082     2373       60    53248        0             0 bash
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [ 886086]     0 886086     2969       68    69632        0             0 systemctl
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=885868,uid=111
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: Out of memory: Killed process 885868 (java) total-vm:12537400kB, anon-rss:8726356kB, file-rss:0kB, shmem-rss:0kB, UID:111 pgtables:17572kB oom_score_adj:0
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=193.27.228.58 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=20932 PROTO=TCP SPT=50633 DPT=>
Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom_reaper: reaped process 885868 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit elasticsearch.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 9.
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit elasticsearch.service has entered the 'failed' state with result 'signal'.
Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has finished with a failure.
--
-- The job identifier is 419853 and the job result is failed.
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: (CRON) info (No MTA installed, discarding output)
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: (CRON) info (No MTA installed, discarding output)
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: pam_unix(cron:session): session closed for user root
Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: pam_unix(cron:session): session closed for user root
Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886113]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886114]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh)
Jul 13 15:28:01 vmi503579.contaboserver.net systemd[1]: Starting Elasticsearch...
-- Subject: A start job for unit elasticsearch.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has begun execution.
--
-- The job identifier is 419985.
Jul 13 15:28:08 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=26420 PROTO=TCP SPT=59598 DPT>
Jul 13 15:28:30 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=60186 PROTO=TCP SPT=59598 DPT>
Jul 13 15:28:53 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=47.100.237.187 DST=173.249.58.30 LEN=40 TOS=0x14 PREC=0x00 TTL=241 ID=33623 PROTO=TCP SPT=42745 DPT>
Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886403]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886404]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh)
Jul 13 15:29:21 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=185.53.90.85 DST=173.249.58.30 LEN=58 TOS=0x08 PREC=0x00 TTL=246 ID=54321 PROTO=UDP SPT=44853 DPT=5>
lines 1157-1213/1213 (END)

The log say out of memory but I dont know which memory.
command df returns this

root@vmi503579:~# df
Filesystem     1K-blocks      Used Available Use% Mounted on
udev             8170104         0   8170104   0% /dev
tmpfs            1639756       952   1638804   1% /run
/dev/sda2      410835424 160177256 229719160  42% /
tmpfs            8198768         0   8198768   0% /dev/shm
tmpfs               5120         0      5120   0% /run/lock
tmpfs            8198768         0   8198768   0% /sys/fs/cgroup
/dev/loop2         56832     56832         0 100% /snap/core18/2074
/dev/loop4         56832     56832         0 100% /snap/core18/2066
/dev/loop5         63232     63232         0 100% /snap/core20/1026
/dev/loop0         43264     43264         0 100% /snap/certbot/1201
/dev/loop6        224256    224256         0 100% /snap/gnome-3-34-1804/66
/dev/loop7         66688     66688         0 100% /snap/gtk-common-themes/1515
/dev/loop3         63232     63232         0 100% /snap/core20/975
/dev/loop8         33152     33152         0 100% /snap/snapd/12398
/dev/loop9         33152     33152         0 100% /snap/snapd/12159
/dev/loop10       224256    224256         0 100% /snap/gnome-3-34-1804/72
/dev/sda1         944120     75396    803548   9% /boot
tmpfs            1639752         0   1639752   0% /run/user/110
/dev/loop11        43392     43392         0 100% /snap/certbot/1280
tmpfs            1639752         0   1639752   0% /run/user/0

Is there somebody who understand it? Thanks for any help.

The OOM killer from the operating system kicks in if there is not enough memory for all processes available. Usually (not always) when not enough memory is available, the process with the most memory being used gets killed, which was Elasticsearch at that point. However this means it was not the only process requiring memory.

Are there other processes running, that need a lot of memory?

Are you running only Elasticsearch on that node or other services as well?

Hmm yes server is on its limits. It runs automated tests with parallel Chrome instances. So this is the problem with resources and i need more memory. Am I right?

yes, please isolate such processes properly from each other, so that only those processes get killed, that really take all that memory unexpectedly - otherwise it will be hard to track down the real culprit.

How can I isolate the processes? What is it? They run as crons.

You need to run them on different systems or in own containers or limit their memory usage in your existing systems. This is not an Elasticsearch problem, but an operating system one though.

From an Elasticsearch perspective I would try to only run one service per system, so only Elasticsearch on this one, and your cron jobs on another.

Thanks a lot. May be I will stop the tests.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.