Logstash out of memory error

I have logstash in a docker container that crashes and says out of memory error after restart. I restart it using docker-compose restart logstash. docker stats says it consumes 400MiB~ of RAM when it's running normally and free -m says that I have ~600 available when it crashes. They are on a 2GB RAM host. The work around I discovered for this was to docker-compose down and docker-compose up. Need help determining why as this hinders automation on updating logstash.

Please show the exact error message.

logstash_1       | Memory: 4k page, physical 2048184k(71608k free), swap 0k(0k free)
logstash_1       | 
logstash_1       | vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0
logstash_1       | 
logstash_1       | time: Fri Oct 20 13:13:49 2017
logstash_1       | elapsed time: 0 seconds (0d 0h 0m 0s)
logstash_1       | 
logstash_1       | #
logstash_1       | # There is insufficient memory for the Java Runtime Environment to continue.
logstash_1       | # Native memory allocation (mmap) failed to map 899284992 bytes for committing reserved memory.
logstash_1       | OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000ca660000, 899284992, 0) failed; error='Out of memory' (errno=12)
logstash_1       | # Can not save log file, dump to screen..
logstash_1       | #
logstash_1       | # There is insufficient memory for the Java Runtime Environment to continue.
logstash_1       | # Native memory allocation (mmap) failed to map 899284992 bytes for committing reserved memory.
logstash_1       | # Possible reasons:
logstash_1       | #   The system is out of physical RAM or swap space
logstash_1       | #   In 32 bit mode, the process size limit was hit
logstash_1       | # Possible solutions:
logstash_1       | #   Reduce memory load on the system
logstash_1       | #   Increase physical memory or swap space
logstash_1       | #   Check if swap backing store is full
logstash_1       | #   Use 64 bit Java on a 64 bit OS
logstash_1       | #   Decrease Java heap size (-Xmx/-Xms)
logstash_1       | #   Decrease number of Java threads
logstash_1       | #   Decrease Java thread stack sizes (-Xss)
logstash_1       | #   Set larger code cache with -XX:ReservedCodeCacheSize=
logstash_1       | # This output file may be truncated or incomplete.
logstash_1       | #
logstash_1       | #  Out of Memory Error (os_linux.cpp:2658), pid=8, tid=0x00007fafc1868ae8
logstash_1       | #
logstash_1       | # JRE version:  (8.0_131-b11) (build )
logstash_1       | # Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
logstash_1       | # Derivative: IcedTea 3.4.0
logstash_1       | # Distribution: Custom build (Fri Jun 16 13:41:54 GMT 2017)
logstash_1       | # Core dump written. Default location: /medikus/core or core.8
logstash_1       | #
logstash_1       | 
logstash_1       | ---------------  T H R E A D  ---------------
logstash_1       | 
logstash_1       | Current thread (0x000055d8a88a7800):  JavaThread "Unknown thread" [_thread_in_vm, id=25, stack(0x00007fafc1668000,0x00007fafc1868aa8)]
logstash_1       | 
logstash_1       | Stack: [0x00007fafc1668000,0x00007fafc1868aa8]
logstash_1       | 
logstash_1       | ---------------  P R O C E S S  ---------------
logstash_1       | 
logstash_1       | Java Threads: ( => current thread )
logstash_1       | 
logstash_1       | Other Threads:
logstash_1       | 
logstash_1       | =>0x000055d8a88a7800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=25, stack(0x00007fafc1668000,0x00007fafc1868aa8)]
logstash_1       | 
logstash_1       | VM state:not at safepoint (not fully initialized)
logstash_1       | 
logstash_1       | VM Mutex/Monitor currently owned by a thread: None
logstash_1       | 
logstash_1       | GC Heap History (0 events):
logstash_1       | No events
logstash_1       | 
logstash_1       | Deoptimization events (0 events):
logstash_1       | No events
logstash_1       | 
logstash_1       | Internal exceptions (0 events):
logstash_1       | No events
logstash_1       | 
logstash_1       | Events (0 events):
logstash_1       | No events
logstash_1       | Dynamic libraries:
logstash_1       | c0000000-ca660000 rw-p 00000000 00:00 0 
logstash_1       | 55d8a68fb000-55d8a68fc000 r-xp 00000000 00:36 13575                      /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1       | 55d8a6afb000-55d8a6afc000 r--p 00000000 00:36 13575                      /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1       | 55d8a6afc000-55d8a6afd000 rw-p 00001000 00:36 13575                      /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
logstash_1       | 55d8a8890000-55d8a88c0000 rw-p 00000000 00:00 0                          [heap]
logstash_1       | 7fafb175a000-7fafb19b0000 rw-p 00000000 00:00 0 
logstash_1       | 7fafb19b0000-7fafb1b5c000 ---p 00000000 00:00 0 
logstash_1       | 7fafb1b5c000-7fafb1b5d000 rw-p 00000000 00:00 0 
logstash_1       | 7fafb1b5d000-7fafb1b5e000 ---p 00000000 00:00 0 
logstash_1       | 7fafb1b5e000-7fafb1c69000 rw-p 00000000 00:00 0 
logstash_1       | 7fafb1c69000-7fafb201f000 ---p 00000000 00:00 0 
logstash_1       | 7fafb201f000-7fafb228f000 rwxp 00000000 00:00 0 
logstash_1       | 7fafb228f000-7fafc101f000 ---p 00000000 00:00 0 
logstash_1       | 7fafc101f000-7fafc1226000 r-xp 00000000 00:36 13607                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1       | 7fafc1226000-7fafc1227000 r--p 00007000 00:36 13607                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1       | 7fafc1227000-7fafc1228000 rw-p 00008000 00:36 13607                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
logstash_1       | 7fafc1228000-7fafc1454000 r-xp 00000000 00:36 13593                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1       | 7fafc1454000-7fafc1455000 r--p 0002c000 00:36 13593                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1       | 7fafc1455000-7fafc1456000 rw-p 0002d000 00:36 13593                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
logstash_1       | 7fafc1456000-7fafc1457000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc1457000-7fafc1663000 r-xp 00000000 00:36 13602                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1       | 7fafc1663000-7fafc1665000 r--p 0000c000 00:36 13602                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1       | 7fafc1665000-7fafc1666000 rw-p 0000e000 00:36 13602                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
logstash_1       | 7fafc1666000-7fafc1668000 ---p 00000000 00:00 0 
logstash_1       | 7fafc1668000-7fafc166b000 ---p 00000000 00:00 0 
logstash_1       | 7fafc166b000-7fafc1869000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc1869000-7fafc1a79000 r-xp 00000000 00:36 13601                      /usr/lib/libgcc_s.so.1
logstash_1       | 7fafc1a79000-7fafc1a7a000 r--p 00010000 00:36 13601                      /usr/lib/libgcc_s.so.1
logstash_1       | 7fafc1a7a000-7fafc1a7b000 rw-p 00011000 00:36 13601                      /usr/lib/libgcc_s.so.1
logstash_1       | 7fafc1a7b000-7fafc1dbb000 r-xp 00000000 00:36 13600                      /usr/lib/libstdc++.so.6.0.22
logstash_1       | 7fafc1dbb000-7fafc1dc9000 r--p 00140000 00:36 13600                      /usr/lib/libstdc++.so.6.0.22
logstash_1       | 7fafc1dc9000-7fafc1dca000 rw-p 0014e000 00:36 13600                      /usr/lib/libstdc++.so.6.0.22
logstash_1       | 7fafc1dca000-7fafc1dcd000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc1dcd000-7fafc27e4000 r-xp 00000000 00:36 13596                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1       | 7fafc27e4000-7fafc2879000 r--p 00817000 00:36 13596                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1       | 7fafc2879000-7fafc28a1000 rw-p 008ac000 00:36 13596                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
logstash_1       | 7fafc28a1000-7fafc28d1000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc28d1000-7fafc2ae6000 r-xp 00000000 00:36 13592                      /lib/libz.so.1.2.11
logstash_1       | 7fafc2ae6000-7fafc2ae7000 r--p 00015000 00:36 13592                      /lib/libz.so.1.2.11
logstash_1       | 7fafc2ae7000-7fafc2ae8000 rw-p 00016000 00:36 13592                      /lib/libz.so.1.2.11
logstash_1       | 7fafc2ae8000-7fafc2cf4000 r-xp 00000000 00:36 13590                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1       | 7fafc2cf4000-7fafc2cf5000 r--p 0000c000 00:36 13590                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1       | 7fafc2cf5000-7fafc2cf6000 rw-p 0000d000 00:36 13590                      /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
logstash_1       | 7fafc2cf6000-7fafc2d7f000 r-xp 00000000 00:36 29                         /lib/ld-musl-x86_64.so.1
logstash_1       | 7fafc2e72000-7fafc2e73000 ---p 00000000 00:00 0 
logstash_1       | 7fafc2e73000-7fafc2f74000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc2f74000-7fafc2f7c000 rw-s 00000000 00:36 13606                      /tmp/hsperfdata_logstash/8
logstash_1       | 7fafc2f7c000-7fafc2f7d000 rw-p 00000000 00:00 0 
logstash_1       | 7fafc2f7d000-7fafc2f7e000 r--p 00000000 00:00 0 
logstash_1       | 7fafc2f7e000-7fafc2f7f000 r--p 00088000 00:36 29                         /lib/ld-musl-x86_64.so.1
logstash_1       | 7fafc2f7f000-7fafc2f80000 rw-p 00089000 00:36 29                         /lib/ld-musl-x86_64.so.1
logstash_1       | 7fafc2f80000-7fafc2f83000 rw-p 00000000 00:00 0 
logstash_1       | 7ffea64fa000-7ffea651c000 rw-p 00000000 00:00 0                          [stack]
logstash_1       | 7ffea65ce000-7ffea65d0000 r--p 00000000 00:00 0                          [vvar]
logstash_1       | 7ffea65d0000-7ffea65d2000 r-xp 00000000 00:00 0                          [vdso]
logstash_1       | ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
logstash_1       | VM Arguments:
logstash_1       | jvm_args: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Xmx1g -Xms1g -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh 
logstash_1       | java_command: org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -f /etc/logstash/conf.d/ --config.reload.automatic true --log.level error
logstash_1       | java_class_path (initial): :
logstash_1       | Launcher Type: SUN_STANDARD
logstash_1       | 
logstash_1       | Environment Variables:
logstash_1       | PATH=/usr/share/logstash/bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
logstash_1       | LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64
logstash_1       | 
logstash_1       | Signal Handlers:
logstash_1       | SIGSEGV: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGBUS: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGFPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGPIPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGXFSZ: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGILL: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1       | SIGUSR2: [libjvm.so+0x56014b], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
logstash_1       | SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1       | SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1       | SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1       | SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
logstash_1       | 
logstash_1       | 
logstash_1       | ---------------  S Y S T E M  ---------------
logstash_1       | 
logstash_1       | OS:NAME="Alpine Linux"
logstash_1       | ID=alpine
logstash_1       | VERSION_ID=3.6.2
logstash_1       | PRETTY_NAME="Alpine Linux v3.6"
logstash_1       | HOME_URL="http://alpinelinux.org"
logstash_1       | BUG_REPORT_URL="http://bugs.alpinelinux.org"
logstash_1       | 
logstash_1       | uname:Linux 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64
logstash_1       | libc:glibc 2.9 NPTL 
logstash_1       | rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
logstash_1       | load average:0.94 0.43 0.27
logstash_1       | 
logstash_1       | /proc/meminfo:
logstash_1       | MemTotal:        2048184 kB
logstash_1       | MemFree:           70364 kB
logstash_1       | MemAvailable:     638492 kB
logstash_1       | Buffers:           61608 kB
logstash_1       | Cached:           590360 kB
logstash_1       | SwapCached:            0 kB
logstash_1       | Active:          1258408 kB
logstash_1       | Inactive:         477228 kB
logstash_1       | Active(anon):    1091840 kB
logstash_1       | Inactive(anon):    29360 kB
logstash_1       | Active(file):     166568 kB
logstash_1       | Inactive(file):   447868 kB
logstash_1       | Unevictable:        3660 kB
logstash_1       | Mlocked:            3660 kB
logstash_1       | SwapTotal:             0 kB
logstash_1       | SwapFree:              0 kB
logstash_1       | Dirty:              3812 kB
logstash_1       | Writeback:             0 kB
logstash_1       | AnonPages:       1087320 kB
logstash_1       | Mapped:           128980 kB
logstash_1       | Shmem:             35172 kB
logstash_1       | Slab:             186848 kB
logstash_1       | SReclaimable:     141672 kB
logstash_1       | SUnreclaim:        45176 kB
logstash_1       | KernelStack:        7664 kB
logstash_1       | PageTables:         9728 kB
logstash_1       | NFS_Unstable:          0 kB
logstash_1       | Bounce:                0 kB
logstash_1       | WritebackTmp:          0 kB
logstash_1       | CommitLimit:     1024092 kB
logstash_1       | Committed_AS:    3562804 kB
logstash_1       | VmallocTotal:   34359738367 kB
logstash_1       | VmallocUsed:           0 kB
logstash_1       | VmallocChunk:          0 kB
logstash_1       | HardwareCorrupted:     0 kB
logstash_1       | AnonHugePages:    657408 kB
logstash_1       | CmaTotal:              0 kB
logstash_1       | CmaFree:               0 kB
logstash_1       | HugePages_Total:       0
logstash_1       | HugePages_Free:        0
logstash_1       | HugePages_Rsvd:        0
logstash_1       | HugePages_Surp:        0
logstash_1       | Hugepagesize:       2048 kB
logstash_1       | DirectMap4k:      135148 kB
logstash_1       | DirectMap2M:     1961984 kB
logstash_1       | DirectMap1G:           0 kB
logstash_1       | CPU:total 2 (initial active 2) (1 cores per cpu, 1 threads per core) family 6 model 62 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, tsc
logstash_1       | 
logstash_1       | /proc/cpuinfo:
logstash_1       | processor	: 0
logstash_1       | vendor_id	: GenuineIntel
logstash_1       | cpu family	: 6
logstash_1       | model		: 62
logstash_1       | model name	: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
logstash_1       | stepping	: 4
logstash_1       | microcode	: 0x1
logstash_1       | cpu MHz		: 2399.998
logstash_1       | cache size	: 15360 KB
logstash_1       | physical id	: 0
logstash_1       | siblings	: 1
logstash_1       | core id		: 0
logstash_1       | cpu cores	: 1
logstash_1       | apicid		: 0
logstash_1       | initial apicid	: 0
logstash_1       | fpu		: yes
logstash_1       | fpu_exception	: yes
logstash_1       | cpuid level	: 13
logstash_1       | wp		: yes
logstash_1       | flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
logstash_1       | bugs		:
logstash_1       | bogomips	: 4799.99
logstash_1       | clflush size	: 64
logstash_1       | cache_alignment	: 64
logstash_1       | address sizes	: 40 bits physical, 48 bits virtual
logstash_1       | power management:
logstash_1       | 
logstash_1       | processor	: 1
logstash_1       | vendor_id	: GenuineIntel
logstash_1       | cpu family	: 6
logstash_1       | model		: 62
logstash_1       | model name	: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
logstash_1       | stepping	: 4
logstash_1       | microcode	: 0x1
logstash_1       | cpu MHz		: 2399.998
logstash_1       | cache size	: 15360 KB
logstash_1       | physical id	: 1
logstash_1       | siblings	: 1
logstash_1       | core id		: 0
logstash_1       | cpu cores	: 1
logstash_1       | apicid		: 1
logstash_1       | initial apicid	: 1
logstash_1       | fpu		: yes
logstash_1       | fpu_exception	: yes
logstash_1       | cpuid level	: 13
logstash_1       | wp		: yes
logstash_1       | flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
logstash_1       | bugs		:
logstash_1       | bogomips	: 4799.99
logstash_1       | clflush size	: 64
logstash_1       | cache_alignment	: 64
logstash_1       | address sizes	: 40 bits physical, 48 bits virtual
logstash_1       | power management:

Looks like 2 GB RAM and zero swap isn't enough to give your Logstash JVM a 1 GB heap.

So I should I lower it down to 512mb? Is it the same as elasticsearch? Because I did pass in the environment variable ES_JAVA_OPTS: -Xms512m -Xmx512m. Also but why does it only happen during restarts as I mentioned that it runs fine initially.

So I should I lower it down to 512mb?

Well, lower it to whatever is needed for it to run reliably.

Is it the same as elasticsearch?

I'm not sure, check the docs. I believe recent Logstash releases have a jvm.options configuration file.

I'll check and try it out. Do you know how could I check the heap size? So that I could verify if it was set.

Doesn't the JVM process arguments include that information (ps aux | grep java)?

1 Like

Setting the JVM heap to a lower value resolves the issue so far.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.