Hi Team,
I am using logstash 7.4.0 version and getting memory leak errors while staring of logstash but logs are successfully sending to ES and able to view them in Kibana. I am getting memory leak errors only on Prod environment which generates huge logs.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
2020-03-05 05:12:14
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+4-alpine-r1 mixed mode):
Threads class SMR info:
_java_thread_list=0x0000560e71f506e0, length=10, elements={
0x0000560e6f457000, 0x0000560e6f5bd800, 0x0000560e6f5c1800, 0x0000560e6f5ee000,
0x0000560e6f5dc000, 0x0000560e6f5de000, 0x0000560e6f5e0800, 0x0000560e6f768000,
0x0000560e6f76b800, 0x0000560e71f4f000
}
"main" #1 prio=5 os_prio=0 cpu=11535.92ms elapsed=15.30s tid=0x0000560e6f457000 nid=0x13 runnable [0x00007f22d6da1000]
java.lang.Thread.State: RUNNABLE
at org.jruby.org.objectweb.asm.SymbolTable.addConstantNameAndType(SymbolTable.java:740)
at org.jruby.org.objectweb.asm.SymbolTable.addConstantMemberReference(SymbolTable.java:572)
at org.jruby.ir.Compiler$1.load(Compiler.java:89)
at org.jruby.Ruby.runScript(Ruby.java:854)
at org.jruby.Ruby.runNormally(Ruby.java:777)
at org.jruby.Ruby.runNormally(Ruby.java:795)
at org.jruby.Ruby.runFromMain(Ruby.java:607)
at org.logstash.Logstash.run(Logstash.java:102)
at org.logstash.Logstash.main(Logstash.java:45)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=10.56ms elapsed=15.26s tid=0x0000560e6f5bd800 nid=0x17 waiting on condition [0x00007f22ad7fe000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.4/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@11.0.4/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.4/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=3.01ms elapsed=15.26s tid=0x0000560e6f5c1800 nid=0x18 in Object.wait() [0x00007f22ad6fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.4/Native Method)
- waiting on <0x000000050b291ad8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.4/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x000000050b291ad8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.4/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.4/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.53ms elapsed=15.25s tid=0x0000560e6f5ee000 nid=0x19 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=11090.81ms elapsed=15.25s tid=0x0000560e6f5dc000 nid=0x1a runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Compiling: 10055 4 org.jruby.ir.dataflow.analyses.LiveVariableNode::markDeadInstructions (447 bytes)
"C1 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=5358.31ms elapsed=15.25s tid=0x0000560e6f5de000 nid=0x1b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #7 daemon prio=9 os_prio=0 cpu=247.47ms elapsed=15.26s tid=0x0000560e6f5e0800 nid=0x1c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 cpu=0.12ms elapsed=15.22s tid=0x0000560e6f768000 nid=0x1d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=12.22ms elapsed=15.21s tid=0x0000560e6f76b800 nid=0x1f in Object.wait() [0x00007f22acff4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.4/Native Method)
- waiting on <0x000000050aa796d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.4/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x000000050aa796d8> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@11.0.4/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base@11.0.4/InnocuousThread.java:134)
"Attach Listener" #14 daemon prio=9 os_prio=0 cpu=0.65ms elapsed=5.50s tid=0x0000560e71f4f000 nid=0x39 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 cpu=189.56ms elapsed=15.27s tid=0x0000560e6f5b2800 nid=0x16 runnable
"GC Thread#0" os_prio=0 cpu=492.37ms elapsed=15.30s tid=0x0000560e6f47b000 nid=0x14 runnable
"GC Thread#1" os_prio=0 cpu=463.40ms elapsed=12.28s tid=0x0000560e70609800 nid=0x20 runnable
"CMS Main Thread" os_prio=0 cpu=862.90ms elapsed=15.30s tid=0x0000560e6f4ab000 nid=0x15 runnable
"VM Periodic Task Thread" os_prio=0 cpu=15.03ms elapsed=15.22s tid=0x0000560e6f757800 nid=0x1e waiting on condition
JNI global refs: 30, weak refs: 0
Heap
par new generation total 153344K, used 119627K [0x0000000500000000, 0x000000050a660000, 0x000000050a660000)
eden space 136320K, 75% used [0x0000000500000000, 0x0000000506432f18, 0x0000000508520000)
from space 17024K, 100% used [0x00000005095c0000, 0x000000050a660000, 0x000000050a660000)
to space 17024K, 0% used [0x0000000508520000, 0x0000000508520000, 0x00000005095c0000)
concurrent mark-sweep generation total 4023936K, used 57118K [0x000000050a660000, 0x0000000600000000, 0x0000000800000000)
Metaspace used 66460K, capacity 73714K, committed 73820K, reserved 1110016K
class space used 10587K, capacity 12684K, committed 12736K, reserved 1048576K
21:12:23.885 [Converge PipelineAction::Create] INFO org.reflections.Reflections - Reflections took 40 ms to scan 1 urls, producing 20 keys and 40 values
========================================================================