Hi,
I created a simple logstash pipeline on Windows with a stdin and a webhdfs output (with kerberos gssapi and a keytab) and I am able to send a message which is received and stored properly by the hdfs namenode.
But logstash is unstable and crash almost systematically after one or few messages.
The crash is really low level (ntdll.dll) after jruby calls.
I tested few JRE 1.8 version : 32-bit, 64-bit, Client VM, Server VM without success.
I am stuck on this so I'am throwing a message in a bottle.
If you have any ideas to guide me...
Maybe a specific JVM version or setting to avoid this ?
Thanks!!
Note that logstash works well on this server with other classic pipelines (elasticsearch output).
Here s the stack for the 32-bit client VM JRE run (truncated due to body size limitation) :
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x77b7e43e, pid=4036, tid=0x00000e4c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) Client VM (25.131-b11 mixed mode windows-x86 )
# Problematic frame:
# C [ntdll.dll+0x2e43e]
--------------- T H R E A D ---------------
Current thread (0x4947c000): JavaThread "Ruby-0-JIT-1" daemon [_thread_in_vm, id=3660, stack(0x49d40000,0x49f40000)]
...
Stack: [0x49d40000,0x49f40000], sp=0x49f3eb6c, free space=2042k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [ntdll.dll+0x2e43e]
C [ntdll.dll+0x2e0a3]
C [kernel32.dll+0x114ad]
C [MSVCR100.dll+0x1016a]
V [jvm.dll+0x162b73]
V [jvm.dll+0x48ddf]
V [jvm.dll+0x491c2]
V [jvm.dll+0x50ed4]
V [jvm.dll+0x51fc0]
V [jvm.dll+0x4ea5a]
V [jvm.dll+0x39e77]
V [jvm.dll+0x5045f]
V [jvm.dll+0x107f59]
V [jvm.dll+0x10812b]
C [java.dll+0x1c7d]
C 0x49f3f5e0
j org.jruby.util.ClassCache.defineClass(Lorg/jruby/util/ClassCache$ClassGenerator;)Ljava/lang/Class;+31
j org.jruby.util.ClassCache.cacheClassByKey(Ljava/lang/Object;Lorg/jruby/util/ClassCache$ClassGenerator;)Ljava/lang/Class;+108
j org.jruby.compiler.JITCompiler$JITTask.run()V+328
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask.run()V+42
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [jvm.dll+0x15af45]
V [jvm.dll+0x2213ae]
V [jvm.dll+0x15afde]
V [jvm.dll+0x15b166]
V [jvm.dll+0x15b1d7]
V [jvm.dll+0xff36f]
V [jvm.dll+0x17dc30]
V [jvm.dll+0x17e4aa]
V [jvm.dll+0x1c2ec6]
C [MSVCR100.dll+0x5c556]
C [MSVCR100.dll+0x5c600]
C [kernel32.dll+0x1336a]
C [ntdll.dll+0x39902]
C [ntdll.dll+0x398d5]
C 0x00000000
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 4693 java.lang.ClassLoader.defineClass1(Ljava/lang/String;[BIILjava/security/ProtectionDomain;Ljava/lang/String;)Ljava/lang/Class; (0 bytes) @ 0x027a1336 [0x027a1290+0xa6]
J 4689 C1 java.lang.ClassLoader.defineClass(Ljava/lang/String;[BIILjava/security/ProtectionDomain;)Ljava/lang/Class; (43 bytes) @ 0x027a3333 [0x027a3290+0xa3]
j org.jruby.util.ClassCache$OneShotClassLoader.defineClass(Ljava/lang/String;[B)Ljava/lang/Class;+9
j org.jruby.util.ClassCache.defineClass(Lorg/jruby/util/ClassCache$ClassGenerator;)Ljava/lang/Class;+31
j org.jruby.util.ClassCache.cacheClassByKey(Ljava/lang/Object;Lorg/jruby/util/ClassCache$ClassGenerator;)Ljava/lang/Class;+108
j org.jruby.compiler.JITCompiler$JITTask.run()V+328
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask.run()V+42
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
...
Internal exceptions (10 events):
Event: 44.219 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04981b28) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.221 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x049838c8) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.244 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04a1f4b8) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.244 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04a210f8) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.244 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04a243a8) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.244 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04a25df8) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.245 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x04a28768) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.284 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x046ae2a0) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
Event: 44.285 Thread 0x4b492800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x046b0140) thrown at [C:\re\workspace\8-2-build-windows-i586-cygwin\jdk8u131\8869\hotspot\src\share\vm\runtime\sharedRuntime.cpp, line 605]
...
--------------- S Y S T E M ---------------
OS: Windows Server 2008 R2 , 64 bit Build 7601 (6.1.7601.23796)
CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, tsc, tscinvbit
Memory: 4k page, physical 4193848k(866656k free), swap 8385836k(5426316k free)
vm_info: Java HotSpot(TM) Client VM (25.131-b11) for windows-x86 JRE (1.8.0_131-b11), built on Mar 15 2017 01:26:29 by "java_re" with MS VC++ 10.0 (VS2010)