# ES 6.4.3 docker container keep crash with error code 139 **URL:** \ **Category:** Elasticsearch\ **Created:** [January 17, 2019, 4:54pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684 "2019-01-17T16:54:40Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 17, 2019, 4:54pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/1 "2019-01-17T16:54:40Z")
Hi Everyone, I met a problem with ES v6.4.3 official docker container running on CentOS 7.6 and searchguard plugin.the docker container can start and working,but it will crash with error code 139,which cause we lost some of the nodes. I perfomed some search against the docker container error code 139 ,it seems that the java program inside the container is crashed and generated a core-dump. Does anyone can help on this? Thanks a lot! ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 17, 2019, 5:08pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/2 "2019-01-17T17:08:18Z")
It's most probably a memory issue. Have you tried to increase the memory available to the Docker container? and then maybe the heap for the JVM inside the Docker container as well? ---
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 18, 2019, 1:10pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/3 "2019-01-18T13:10:33Z")
our current configuration for an instance is about 31G memory and I thinks this is enouth as we do not have much workload so far ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5ffa2aff1639b9980d01fff8b13b9a53e6850080.png) The following is the current configuration in docker-compose.yml: elasticsearch-instance1: privileged: true image: [docker.elastic.co/elasticsearch/elasticsearch:6.4.3](http://docker.elastic.co/elasticsearch/elasticsearch:6.4.3) container\_name: elasticsearch-instance1 environment: - cluster.name=njmonitor\_cluster - bootstrap.memory\_lock=true - "ES\_JAVA\_OPTS=-Xms31g -Xmx31g" - xpack.monitoring.collection.enabled=true ulimits: memlock: soft: -1 hard: -1 ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2019, 1:12pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/4 "2019-01-18T13:12:45Z")
Indeed, that seems like enough heap. Are there any logs from ES that you can share? ---
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 18, 2019, 1:19pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/5 "2019-01-18T13:19:50Z")
There is a core file and a hs\_err\_pid1.log -rwxrwxrwx 1 root root 37001433088 Jan 15 10:59 core.1 -rwxrwxrwx 1 root root 195996 Jan 15 11:00 hs\_err\_pid1.log some information in the he hs\_err\_pid1.log: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fea1ca2af91, pid=1, tid=694 # # JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13) # Java VM: OpenJDK 64-Bit Server VM (10.0.2+13, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64) # Problematic frame: # J 19317 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fea1ca2af91 [0x00007fea1ca2a560+0x0000000000000a31] # # Core dump will be written. Default location: /usr/share/elasticsearch/core.1 # # If you would like to submit a bug report, please visit: # [http://bugreport.java.com/bugreport/crash.jsp](http://bugreport.java.com/bugreport/crash.jsp) # --------------- S U M M A R Y ------------ Command Line: -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Dj ava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimizati on=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMem oryError -Des.cgroups.hierarchy.override=/ -Xms31g -Xmx31g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distri bution.flavor=default -Des.distribution.type=tar org.elasticsearch.bootstrap.Elasticsearch -Expack.monitoring.collection.enabled=true -Ecluster.name=njmammot h -Ebootstrap.memory\_lock=true Host: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz, 56 cores, 125G, CentOS Linux release 7.5.1804 (Core) Time: Tue Jan 15 01:07:59 2019 UTC elapsed time: 3216 seconds (0d 0h 53m 36s) --------------- T H R E A D --------------- Current thread (0x00007fe3dc001800): JavaThread "elasticsearch[172.18.140.8-instance2][[.monitoring-es-6-2019.01.15][0]: Lucene Merge Thread #45]" daemon [\_ thread\_in\_Java, id=694, stack(0x00007fe68cdfb000,0x00007fe68cefc000)] Stack: [0x00007fe68cdfb000,0x00007fe68cefc000], sp=0x00007fe68cefa1d8, free space=1020k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) J 19317 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fea1ca2af91 [0x00007fea1ca2a560+0x0000000000000a31] [error occurred during error reporting (printing native stack), id 0xb] siginfo: si\_signo: 11 (SIGSEGV), si\_code: 2 (SEGV\_ACCERR), si\_addr: 0x00000010001fffc6 --------------- S Y S T E M --------------- OS:CentOS Linux release 7.5.1804 (Core) uname:Linux 3.10.0-957.el7.x86\_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86\_64 libc:glibc 2.17 NPTL 2.17 rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity, DATA infinity, FSIZE infinity load average:0.07 0.05 0.05 /proc/meminfo: MemTotal: 131328224 kB MemFree: 55848904 kB MemAvailable: 56911444 kB Buffers: 2104 kB Cached: 1693076 kB SwapCached: 0 kB Active: 5026956 kB Inactive: 798452 kB Active(anon): 4305656 kB Inactive(anon): 10412 kB Active(file): 721300 kB Inactive(file): 788040 kB Unevictable: 67609536 kB Mlocked: 82416968 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 11048 kB Writeback: 0 kB AnonPages: 71740812 kB Mapped: 255768 kB Shmem: 10980 kB Slab: 378204 kB SReclaimable: 124872 kB SUnreclaim: 253332 kB KernelStack: 29616 kB PageTables: 149840 kB NFS\_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 65664112 kB Committed\_AS: 72773404 kB VmallocTotal: 34359738367 kB VmallocUsed: 636292 kB VmallocChunk: 34291798012 kB HardwareCorrupted: 0 kB AnonHugePages: 69806080 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages\_Total: 0 HugePages\_Free: 0 HugePages\_Rsvd: 0 HugePages\_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 339808 kB DirectMap2M: 5476352 kB DirectMap1G: 130023424 kB container (cgroup) information: container\_type: cgroupv1 cpu\_cpuset\_cpus: 0-55 cpu\_memory\_nodes: 0-1 active\_processor\_count: 56 cpu\_quota: -2 cpu\_period: -2 cpu\_shares: -2 memory\_limit\_in\_bytes: -1 memory\_and\_swap\_limit\_in\_bytes: -1 memory\_soft\_limit\_in\_bytes: -1 memory\_usage\_in\_bytes: 36826869760 memory\_max\_usage\_in\_bytes: 36830556160 CPU:total 56 (initial active 56) (14 cores per cpu, 2 threads per core) family 6 model 85 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, s se4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma CPU Model and flags from /proc/cpuinfo: model name : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant\_tsc art arch\_perfmon pebs bts rep\_good nopl xtopology nonstop\_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds\_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4\_1 sse4\_2 x2apic movbe popcnt tsc\_deadline\_timer aes xsave avx f16c rdrand lahf\_lm abm 3dnowprefetch epb cat\_l3 cd p\_l3 intel\_ppin intel\_pt ssbd mba ibrs ibpb stibp tpr\_shadow vnmi flexpriority ept vpid fsgsbase tsc\_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt\_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm\_llc cqm\_occup\_llc cqm\_mbm\_total cqm\_mbm\_local d therm ida arat pln pts hwp\_epp pku ospke spec\_ctrl intel\_stibp flush\_l1d Memory: 4k page, physical 131328224k(55848904k free), swap 0k(0k free) vm\_info: OpenJDK 64-Bit Server VM (10.0.2+13) for linux-amd64 JRE (10.0.2+13), built on Jun 27 2018 17:52:12 by "mach5one" with gcc 4.9.2 ---
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 18, 2019, 1:26pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/6 "2019-01-18T13:26:41Z")
The docker image using CentOS 7.5 version,do We need to use the same version for the host ,now the host is CentOS 7.6 ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2019, 1:40pm UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/7 "2019-01-18T13:40:03Z")
Hmm I see you're running Java 10 inside the container and there's an OpenJDK bug in that version. You're not the first to experience this. There are a few things to try in this thread (namely use -XX:UseAVX=2): [https://github.com/elastic/elasticsearch/issues/31425#issuecomment-402522285](https://github.com/elastic/elasticsearch/issues/31425#issuecomment-402522285) Can you check ? ---
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 19, 2019, 5:50am UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/8 "2019-01-19T05:50:24Z")
Thanks a lot! I will test the option mentioned and feedback the result! ---
**Author:** ![blackgrill](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@blackgrill](https://discuss.elastic.co/u/blackgrill)\ **Post date:** [January 25, 2019, 2:17am UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/9 "2019-01-25T02:17:15Z")
After applied the JVM option,the cluster is stable now.Thanks a lot for your kind support! ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 25, 2019, 8:33am UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/10 "2019-01-25T08:33:00Z")
Fantastic, glad it was helpful! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 22, 2019, 8:33am UTC](https://discuss.elastic.co/t/es-6-4-3-docker-container-keep-crash-with-error-code-139/164684/11 "2019-02-22T08:33:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.