# Can somebody tell me what is in this log? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 13, 2021, 1:34pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547 "2021-07-13T13:34:02Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Vladimir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladimir/32/91042_2.png) [@Vladimir](https://discuss.elastic.co/u/Vladimir)\ **Post date:** [July 13, 2021, 1:34pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/1 "2021-07-13T13:34:02Z")
I can not start Elastic although I can do it few days before. ```auto root@vmi503579:~# journalctl -ex Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [884825] 33 884825 59113 6477 241664 686 0 apache2 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885861] 0 885861 2771 158 57344 11 0 cron Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885862] 0 885862 652 17 40960 0 0 sh Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885863] 0 885863 2373 51 57344 0 0 bash Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885867] 0 885867 2969 70 65536 0 0 systemctl Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885868] 111 885868 3134350 2181589 17993728 0 0 java Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886065] 111 886065 27101 148 86016 0 0 controller Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886080] 0 886080 2771 159 57344 11 0 cron Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886081] 0 886081 652 16 45056 0 0 sh Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886082] 0 886082 2373 60 53248 0 0 bash Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886086] 0 886086 2969 68 69632 0 0 systemctl Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=885868,uid=111 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: Out of memory: Killed process 885868 (java) total-vm:12537400kB, anon-rss:8726356kB, file-rss:0kB, shmem-rss:0kB, UID:111 pgtables:17572kB oom_score_adj:0 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=193.27.228.58 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=20932 PROTO=TCP SPT=50> Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom_reaper: reaped process 885868 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL -- Subject: Unit process exited -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- An ExecStart= process belonging to unit elasticsearch.service has exited. -- -- The process' exit code is 'killed' and its exit status is 9. Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Failed with result 'signal'. -- Subject: Unit failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- The unit elasticsearch.service has entered the 'failed' state with result 'signal'. Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: Failed to start Elasticsearch. -- Subject: A start job for unit elasticsearch.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- A start job for unit elasticsearch.service has finished with a failure. -- -- The job identifier is 419853 and the job result is failed. Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: (CRON) info (No MTA installed, discarding output) Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: (CRON) info (No MTA installed, discarding output) Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: pam_unix(cron:session): session closed for user root Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: pam_unix(cron:session): session closed for user root Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886113]: pam_unix(cron:session): session opened for user root by (uid=0) Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886114]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh) Jul 13 15:28:01 vmi503579.contaboserver.net systemd[1]: Starting Elasticsearch... -- Subject: A start job for unit elasticsearch.service has begun execution -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- A start job for unit elasticsearch.service has begun execution. -- -- The job identifier is 419985. Jul 13 15:28:08 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=26420 PROTO=TCP SPT=5> Jul 13 15:28:30 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=60186 PROTO=TCP SPT=5> Jul 13 15:28:53 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=47.100.237.187 DST=173.249.58.30 LEN=40 TOS=0x14 PREC=0x00 TTL=241 ID=33623 PROTO=TCP SPT=4> Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886403]: pam_unix(cron:session): session opened for user root by (uid=0) Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886404]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh) Jul 13 15:29:21 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=185.53.90.85 DST=173.249.58.30 LEN=58 TOS=0x08 PREC=0x00 TTL=246 ID=54321 PROTO=UDP SPT=448> lines 1157-1213/1213 (END) Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [884825] 33 884825 59113 6477 241664 686 0 apache2 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885861] 0 885861 2771 158 57344 11 0 cron Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885862] 0 885862 652 17 40960 0 0 sh Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885863] 0 885863 2373 51 57344 0 0 bash Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885867] 0 885867 2969 70 65536 0 0 systemctl Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [885868] 111 885868 3134350 2181589 17993728 0 0 java Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886065] 111 886065 27101 148 86016 0 0 controller Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886080] 0 886080 2771 159 57344 11 0 cron Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886081] 0 886081 652 16 45056 0 0 sh Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886082] 0 886082 2373 60 53248 0 0 bash Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [886086] 0 886086 2969 68 69632 0 0 systemctl Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=885868,uid=111 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: Out of memory: Killed process 885868 (java) total-vm:12537400kB, anon-rss:8726356kB, file-rss:0kB, shmem-rss:0kB, UID:111 pgtables:17572kB oom_score_adj:0 Jul 13 15:27:55 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=193.27.228.58 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=20932 PROTO=TCP SPT=50633 DPT=> Jul 13 15:27:55 vmi503579.contaboserver.net kernel: oom_reaper: reaped process 885868 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL -- Subject: Unit process exited -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- An ExecStart= process belonging to unit elasticsearch.service has exited. -- -- The process' exit code is 'killed' and its exit status is 9. Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Failed with result 'signal'. -- Subject: Unit failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- The unit elasticsearch.service has entered the 'failed' state with result 'signal'. Jul 13 15:27:55 vmi503579.contaboserver.net systemd[1]: Failed to start Elasticsearch. -- Subject: A start job for unit elasticsearch.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- A start job for unit elasticsearch.service has finished with a failure. -- -- The job identifier is 419853 and the job result is failed. Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: (CRON) info (No MTA installed, discarding output) Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: (CRON) info (No MTA installed, discarding output) Jul 13 15:27:55 vmi503579.contaboserver.net CRON[885861]: pam_unix(cron:session): session closed for user root Jul 13 15:27:55 vmi503579.contaboserver.net CRON[886080]: pam_unix(cron:session): session closed for user root Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886113]: pam_unix(cron:session): session opened for user root by (uid=0) Jul 13 15:28:01 vmi503579.contaboserver.net CRON[886114]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh) Jul 13 15:28:01 vmi503579.contaboserver.net systemd[1]: Starting Elasticsearch... -- Subject: A start job for unit elasticsearch.service has begun execution -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- A start job for unit elasticsearch.service has begun execution. -- -- The job identifier is 419985. Jul 13 15:28:08 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=26420 PROTO=TCP SPT=59598 DPT> Jul 13 15:28:30 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=89.248.165.203 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=60186 PROTO=TCP SPT=59598 DPT> Jul 13 15:28:53 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=47.100.237.187 DST=173.249.58.30 LEN=40 TOS=0x14 PREC=0x00 TTL=241 ID=33623 PROTO=TCP SPT=42745 DPT> Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886403]: pam_unix(cron:session): session opened for user root by (uid=0) Jul 13 15:29:01 vmi503579.contaboserver.net CRON[886404]: (root) CMD (bash /root/custom-scripts/elasticsearch/start-elasticsearch.sh) Jul 13 15:29:21 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=185.53.90.85 DST=173.249.58.30 LEN=58 TOS=0x08 PREC=0x00 TTL=246 ID=54321 PROTO=UDP SPT=44853 DPT=5> lines 1157-1213/1213 (END) ``` The log say out of memory but I dont know which memory. command df returns this ```auto root@vmi503579:~# df Filesystem 1K-blocks Used Available Use% Mounted on udev 8170104 0 8170104 0% /dev tmpfs 1639756 952 1638804 1% /run /dev/sda2 410835424 160177256 229719160 42% / tmpfs 8198768 0 8198768 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 8198768 0 8198768 0% /sys/fs/cgroup /dev/loop2 56832 56832 0 100% /snap/core18/2074 /dev/loop4 56832 56832 0 100% /snap/core18/2066 /dev/loop5 63232 63232 0 100% /snap/core20/1026 /dev/loop0 43264 43264 0 100% /snap/certbot/1201 /dev/loop6 224256 224256 0 100% /snap/gnome-3-34-1804/66 /dev/loop7 66688 66688 0 100% /snap/gtk-common-themes/1515 /dev/loop3 63232 63232 0 100% /snap/core20/975 /dev/loop8 33152 33152 0 100% /snap/snapd/12398 /dev/loop9 33152 33152 0 100% /snap/snapd/12159 /dev/loop10 224256 224256 0 100% /snap/gnome-3-34-1804/72 /dev/sda1 944120 75396 803548 9% /boot tmpfs 1639752 0 1639752 0% /run/user/110 /dev/loop11 43392 43392 0 100% /snap/certbot/1280 tmpfs 1639752 0 1639752 0% /run/user/0 ``` Is there somebody who understand it? Thanks for any help. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 13, 2021, 1:43pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/2 "2021-07-13T13:43:55Z")
The OOM killer from the operating system kicks in if there is not enough memory for all processes available. Usually (not always) when not enough memory is available, the process with the most memory being used gets killed, which was Elasticsearch at that point. However this means it was not the only process requiring memory. Are there other processes running, that need a lot of memory? Are you running only Elasticsearch on that node or other services as well? ---
**Author:** ![Vladimir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladimir/32/91042_2.png) [@Vladimir](https://discuss.elastic.co/u/Vladimir)\ **Post date:** [July 13, 2021, 2:04pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/3 "2021-07-13T14:04:56Z")
Hmm yes server is on its limits. It runs automated tests with parallel Chrome instances. So this is the problem with resources and i need more memory. Am I right? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 13, 2021, 2:13pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/4 "2021-07-13T14:13:33Z")
yes, please isolate such processes properly from each other, so that only those processes get killed, that really take all that memory unexpectedly - otherwise it will be hard to track down the real culprit. ---
**Author:** ![Vladimir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladimir/32/91042_2.png) [@Vladimir](https://discuss.elastic.co/u/Vladimir)\ **Post date:** [July 13, 2021, 2:30pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/5 "2021-07-13T14:30:53Z")
How can I isolate the processes? What is it? They run as crons. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 13, 2021, 2:53pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/6 "2021-07-13T14:53:32Z")
You need to run them on different systems or in own containers or limit their memory usage in your existing systems. This is not an Elasticsearch problem, but an operating system one though. From an Elasticsearch perspective I would try to **only** run one service per system, so only Elasticsearch on this one, and your cron jobs on another. ---
**Author:** ![Vladimir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladimir/32/91042_2.png) [@Vladimir](https://discuss.elastic.co/u/Vladimir)\ **Post date:** [July 13, 2021, 2:58pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/7 "2021-07-13T14:58:51Z")
Thanks a lot. May be I will stop the tests. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 10, 2021, 2:59pm UTC](https://discuss.elastic.co/t/can-somebody-tell-me-what-is-in-this-log/278547/8 "2021-08-10T14:59:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.