I have logstash in a docker container that crashes and says out of memory error after restart. I restart it using docker-compose restart logstash. docker stats says it consumes 400MiB~ of RAM when it's running normally and free -m says that I have ~600 available when it crashes. They are on a 2GB RAM host. The work around I discovered for this was to docker-compose down and docker-compose up. Need help determining why as this hinders automation on updating logstash.
1 Like
Please show the exact error message.
logstash_1 | Memory: 4k page, physical 2048184k(71608k free), swap 0k(0k free)
logstash_1 |
logstash_1 | vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0
logstash_1 |
logstash_1 | time: Fri Oct 20 13:13:49 2017
logstash_1 | elapsed time: 0 seconds (0d 0h 0m 0s)
logstash_1 |
logstash_1 | #
logstash_1 | # There is insufficient memory for the Java Runtime Environment to continue.
logstash_1 | # Native memory allocation (mmap) failed to map 899284992 bytes for committing reserved memory.
logstash_1 | OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000ca660000, 899284992, 0) failed; error='Out of memory' (errno=12)
logstash_1 | # Can not save log file, dump to screen..
logstash_1 | #
logstash_1 | # There is insufficient memory for the Java Runtime Environment to continue.
logstash_1 | # Native memory allocation (mmap) failed to map 899284992 bytes for committing reserved memory.
logstash_1 | # Possible reasons:
logstash_1 | # The system is out of physical RAM or swap space
logstash_1 | # In 32 bit mode, the process size limit was hit
logstash_1 | # Possible solutions:
logstash_1 | # Reduce memory load on the system
logstash_1 | # Increase physical memory or swap space
logstash_1 | # Check if swap backing store is full
logstash_1 | # Use 64 bit Java on a 64 bit OS
logstash_1 | # Decrease Java heap size (-Xmx/-Xms)
logstash_1 | # Decrease number of Java threads
logstash_1 | # Decrease Java thread stack sizes (-Xss)
logstash_1 | # Set larger code cache with -XX:ReservedCodeCacheSize=
logstash_1 | # This output file may be truncated or incomplete.
logstash_1 | #
logstash_1 | # Out of Memory Error (os_linux.cpp:2658), pid=8, tid=0x00007fafc1868ae8
logstash_1 | #
logstash_1 | # JRE version: (8.0_131-b11) (build )
logstash_1 | # Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
logstash_1 | # Derivative: IcedTea 3.4.0
logstash_1 | # Distribution: Custom build (Fri Jun 16 13:41:54 GMT 2017)
logstash_1 | # Core dump written. Default location: /medikus/core or core.8
logstash_1 | #
logstash_1 |
logstash_1 | --------------- T H R E A D ---------------
logstash_1 |
logstash_1 | Current thread (0x000055d8a88a7800): JavaThread "Unknown thread" [_thread_in_vm, id=25, stack(0x00007fafc1668000,0x00007fafc1868aa8)]
logstash_1 |
logstash_1 | Stack: [0x00007fafc1668000,0x00007fafc1868aa8]
logstash_1 |
logstash_1 | --------------- P R O C E S S ---------------
logstash_1 |
logstash_1 | Java Threads: ( => current thread )
logstash_1 |
logstash_1 | Other Threads:
logstash_1 |
logstash_1 | =>0x000055d8a88a7800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=25, stack(0x00007fafc1668000,0x00007fafc1868aa8)]
logstash_1 |
logstash_1 | VM state:not at safepoint (not fully initialized)
logstash_1 |
logstash_1 | VM Mutex/Monitor currently owned by a thread: None
logstash_1 |
logstash_1 | GC Heap History (0 events):
logstash_1 | No events
logstash_1 |
logstash_1 | Deoptimization events (0 events):
logstash_1 | No events
logstash_1 |
logstash_1 | Internal exceptions (0 events):
logstash_1 | No events
logstash_1 |
logstash_1 | Events (0 events):
logstash_1 | No events
logstash_1 | Dynamic libraries:
logstash_1 | c0000000-ca660000 rw-p 00000000 00:00 0
logstash_1 | 55d8a68fb000-55d8a68fc000 r-xp 00000000 00:36 13575 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1 | 55d8a6afb000-55d8a6afc000 r--p 00000000 00:36 13575 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1 | 55d8a6afc000-55d8a6afd000 rw-p 00001000 00:36 13575 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1 | 55d8a8890000-55d8a88c0000 rw-p 00000000 00:00 0 [heap]
logstash_1 | 7fafb175a000-7fafb19b0000 rw-p 00000000 00:00 0
logstash_1 | 7fafb19b0000-7fafb1b5c000 ---p 00000000 00:00 0
logstash_1 | 7fafb1b5c000-7fafb1b5d000 rw-p 00000000 00:00 0
logstash_1 | 7fafb1b5d000-7fafb1b5e000 ---p 00000000 00:00 0
logstash_1 | 7fafb1b5e000-7fafb1c69000 rw-p 00000000 00:00 0
logstash_1 | 7fafb1c69000-7fafb201f000 ---p 00000000 00:00 0
logstash_1 | 7fafb201f000-7fafb228f000 rwxp 00000000 00:00 0
logstash_1 | 7fafb228f000-7fafc101f000 ---p 00000000 00:00 0
logstash_1 | 7fafc101f000-7fafc1226000 r-xp 00000000 00:36 13607 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1 | 7fafc1226000-7fafc1227000 r--p 00007000 00:36 13607 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1 | 7fafc1227000-7fafc1228000 rw-p 00008000 00:36 13607 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1 | 7fafc1228000-7fafc1454000 r-xp 00000000 00:36 13593 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1 | 7fafc1454000-7fafc1455000 r--p 0002c000 00:36 13593 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1 | 7fafc1455000-7fafc1456000 rw-p 0002d000 00:36 13593 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1 | 7fafc1456000-7fafc1457000 rw-p 00000000 00:00 0
logstash_1 | 7fafc1457000-7fafc1663000 r-xp 00000000 00:36 13602 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1 | 7fafc1663000-7fafc1665000 r--p 0000c000 00:36 13602 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1 | 7fafc1665000-7fafc1666000 rw-p 0000e000 00:36 13602 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1 | 7fafc1666000-7fafc1668000 ---p 00000000 00:00 0
logstash_1 | 7fafc1668000-7fafc166b000 ---p 00000000 00:00 0
logstash_1 | 7fafc166b000-7fafc1869000 rw-p 00000000 00:00 0
logstash_1 | 7fafc1869000-7fafc1a79000 r-xp 00000000 00:36 13601 /usr/lib/libgcc_s.so.1
logstash_1 | 7fafc1a79000-7fafc1a7a000 r--p 00010000 00:36 13601 /usr/lib/libgcc_s.so.1
logstash_1 | 7fafc1a7a000-7fafc1a7b000 rw-p 00011000 00:36 13601 /usr/lib/libgcc_s.so.1
logstash_1 | 7fafc1a7b000-7fafc1dbb000 r-xp 00000000 00:36 13600 /usr/lib/libstdc++.so.6.0.22
logstash_1 | 7fafc1dbb000-7fafc1dc9000 r--p 00140000 00:36 13600 /usr/lib/libstdc++.so.6.0.22
logstash_1 | 7fafc1dc9000-7fafc1dca000 rw-p 0014e000 00:36 13600 /usr/lib/libstdc++.so.6.0.22
logstash_1 | 7fafc1dca000-7fafc1dcd000 rw-p 00000000 00:00 0
logstash_1 | 7fafc1dcd000-7fafc27e4000 r-xp 00000000 00:36 13596 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1 | 7fafc27e4000-7fafc2879000 r--p 00817000 00:36 13596 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1 | 7fafc2879000-7fafc28a1000 rw-p 008ac000 00:36 13596 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1 | 7fafc28a1000-7fafc28d1000 rw-p 00000000 00:00 0
logstash_1 | 7fafc28d1000-7fafc2ae6000 r-xp 00000000 00:36 13592 /lib/libz.so.1.2.11
logstash_1 | 7fafc2ae6000-7fafc2ae7000 r--p 00015000 00:36 13592 /lib/libz.so.1.2.11
logstash_1 | 7fafc2ae7000-7fafc2ae8000 rw-p 00016000 00:36 13592 /lib/libz.so.1.2.11
logstash_1 | 7fafc2ae8000-7fafc2cf4000 r-xp 00000000 00:36 13590 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1 | 7fafc2cf4000-7fafc2cf5000 r--p 0000c000 00:36 13590 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1 | 7fafc2cf5000-7fafc2cf6000 rw-p 0000d000 00:36 13590 /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1 | 7fafc2cf6000-7fafc2d7f000 r-xp 00000000 00:36 29 /lib/ld-musl-x86_64.so.1
logstash_1 | 7fafc2e72000-7fafc2e73000 ---p 00000000 00:00 0
logstash_1 | 7fafc2e73000-7fafc2f74000 rw-p 00000000 00:00 0
logstash_1 | 7fafc2f74000-7fafc2f7c000 rw-s 00000000 00:36 13606 /tmp/hsperfdata_logstash/8
logstash_1 | 7fafc2f7c000-7fafc2f7d000 rw-p 00000000 00:00 0
logstash_1 | 7fafc2f7d000-7fafc2f7e000 r--p 00000000 00:00 0
logstash_1 | 7fafc2f7e000-7fafc2f7f000 r--p 00088000 00:36 29 /lib/ld-musl-x86_64.so.1
logstash_1 | 7fafc2f7f000-7fafc2f80000 rw-p 00089000 00:36 29 /lib/ld-musl-x86_64.so.1
logstash_1 | 7fafc2f80000-7fafc2f83000 rw-p 00000000 00:00 0
logstash_1 | 7ffea64fa000-7ffea651c000 rw-p 00000000 00:00 0 [stack]
logstash_1 | 7ffea65ce000-7ffea65d0000 r--p 00000000 00:00 0 [vvar]
logstash_1 | 7ffea65d0000-7ffea65d2000 r-xp 00000000 00:00 0 [vdso]
logstash_1 | ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
logstash_1 | VM Arguments:
logstash_1 | jvm_args: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Xmx1g -Xms1g -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh
logstash_1 | java_command: org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -f /etc/logstash/conf.d/ --config.reload.automatic true --log.level error
logstash_1 | java_class_path (initial): :
logstash_1 | Launcher Type: SUN_STANDARD
logstash_1 |
logstash_1 | Environment Variables:
logstash_1 | PATH=/usr/share/logstash/bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
logstash_1 | LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64
logstash_1 |
logstash_1 | Signal Handlers:
logstash_1 | SIGSEGV: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGBUS: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGFPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGPIPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGXFSZ: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGILL: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1 | SIGUSR2: [libjvm.so+0x56014b], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1 | SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1 | SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1 | SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1 | SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1 |
logstash_1 |
logstash_1 | --------------- S Y S T E M ---------------
logstash_1 |
logstash_1 | OS:NAME="Alpine Linux"
logstash_1 | ID=alpine
logstash_1 | VERSION_ID=3.6.2
logstash_1 | PRETTY_NAME="Alpine Linux v3.6"
logstash_1 | HOME_URL="http://alpinelinux.org"
logstash_1 | BUG_REPORT_URL="http://bugs.alpinelinux.org"
logstash_1 |
logstash_1 | uname:Linux 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64
logstash_1 | libc:glibc 2.9 NPTL
logstash_1 | rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
logstash_1 | load average:0.94 0.43 0.27
logstash_1 |
logstash_1 | /proc/meminfo:
logstash_1 | MemTotal: 2048184 kB
logstash_1 | MemFree: 70364 kB
logstash_1 | MemAvailable: 638492 kB
logstash_1 | Buffers: 61608 kB
logstash_1 | Cached: 590360 kB
logstash_1 | SwapCached: 0 kB
logstash_1 | Active: 1258408 kB
logstash_1 | Inactive: 477228 kB
logstash_1 | Active(anon): 1091840 kB
logstash_1 | Inactive(anon): 29360 kB
logstash_1 | Active(file): 166568 kB
logstash_1 | Inactive(file): 447868 kB
logstash_1 | Unevictable: 3660 kB
logstash_1 | Mlocked: 3660 kB
logstash_1 | SwapTotal: 0 kB
logstash_1 | SwapFree: 0 kB
logstash_1 | Dirty: 3812 kB
logstash_1 | Writeback: 0 kB
logstash_1 | AnonPages: 1087320 kB
logstash_1 | Mapped: 128980 kB
logstash_1 | Shmem: 35172 kB
logstash_1 | Slab: 186848 kB
logstash_1 | SReclaimable: 141672 kB
logstash_1 | SUnreclaim: 45176 kB
logstash_1 | KernelStack: 7664 kB
logstash_1 | PageTables: 9728 kB
logstash_1 | NFS_Unstable: 0 kB
logstash_1 | Bounce: 0 kB
logstash_1 | WritebackTmp: 0 kB
logstash_1 | CommitLimit: 1024092 kB
logstash_1 | Committed_AS: 3562804 kB
logstash_1 | VmallocTotal: 34359738367 kB
logstash_1 | VmallocUsed: 0 kB
logstash_1 | VmallocChunk: 0 kB
logstash_1 | HardwareCorrupted: 0 kB
logstash_1 | AnonHugePages: 657408 kB
logstash_1 | CmaTotal: 0 kB
logstash_1 | CmaFree: 0 kB
logstash_1 | HugePages_Total: 0
logstash_1 | HugePages_Free: 0
logstash_1 | HugePages_Rsvd: 0
logstash_1 | HugePages_Surp: 0
logstash_1 | Hugepagesize: 2048 kB
logstash_1 | DirectMap4k: 135148 kB
logstash_1 | DirectMap2M: 1961984 kB
logstash_1 | DirectMap1G: 0 kB
logstash_1 | CPU:total 2 (initial active 2) (1 cores per cpu, 1 threads per core) family 6 model 62 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, tsc
logstash_1 |
logstash_1 | /proc/cpuinfo:
logstash_1 | processor : 0
logstash_1 | vendor_id : GenuineIntel
logstash_1 | cpu family : 6
logstash_1 | model : 62
logstash_1 | model name : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
logstash_1 | stepping : 4
logstash_1 | microcode : 0x1
logstash_1 | cpu MHz : 2399.998
logstash_1 | cache size : 15360 KB
logstash_1 | physical id : 0
logstash_1 | siblings : 1
logstash_1 | core id : 0
logstash_1 | cpu cores : 1
logstash_1 | apicid : 0
logstash_1 | initial apicid : 0
logstash_1 | fpu : yes
logstash_1 | fpu_exception : yes
logstash_1 | cpuid level : 13
logstash_1 | wp : yes
logstash_1 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
logstash_1 | bugs :
logstash_1 | bogomips : 4799.99
logstash_1 | clflush size : 64
logstash_1 | cache_alignment : 64
logstash_1 | address sizes : 40 bits physical, 48 bits virtual
logstash_1 | power management:
logstash_1 |
logstash_1 | processor : 1
logstash_1 | vendor_id : GenuineIntel
logstash_1 | cpu family : 6
logstash_1 | model : 62
logstash_1 | model name : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
logstash_1 | stepping : 4
logstash_1 | microcode : 0x1
logstash_1 | cpu MHz : 2399.998
logstash_1 | cache size : 15360 KB
logstash_1 | physical id : 1
logstash_1 | siblings : 1
logstash_1 | core id : 0
logstash_1 | cpu cores : 1
logstash_1 | apicid : 1
logstash_1 | initial apicid : 1
logstash_1 | fpu : yes
logstash_1 | fpu_exception : yes
logstash_1 | cpuid level : 13
logstash_1 | wp : yes
logstash_1 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
logstash_1 | bugs :
logstash_1 | bogomips : 4799.99
logstash_1 | clflush size : 64
logstash_1 | cache_alignment : 64
logstash_1 | address sizes : 40 bits physical, 48 bits virtual
logstash_1 | power management:
Looks like 2 GB RAM and zero swap isn't enough to give your Logstash JVM a 1 GB heap.
2 Likes
So I should I lower it down to 512mb? Is it the same as elasticsearch? Because I did pass in the environment variable ES_JAVA_OPTS: -Xms512m -Xmx512m. Also but why does it only happen during restarts as I mentioned that it runs fine initially.
So I should I lower it down to 512mb?
Well, lower it to whatever is needed for it to run reliably.
Is it the same as elasticsearch?
I'm not sure, check the docs. I believe recent Logstash releases have a jvm.options configuration file.
I'll check and try it out. Do you know how could I check the heap size? So that I could verify if it was set.
Doesn't the JVM process arguments include that information (ps aux | grep java)?
1 Like
Setting the JVM heap to a lower value resolves the issue so far.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.