ES Docker unsafe memory access

(Pablo Ciruzzi) #1


I'm using Docker Compose to start an ES (Version 5.5.3) for development. I followed the guide to do it from here. The problem is that, after some time, I get an error:

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Here is my Docker Compose config:

$ docker-compose config
    container_name: elasticsearch
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      bootstrap.memory_lock: "true" "false"
      xpack.watcher.enabled: "false"
      redo_backend: null
    - 9201:9200/tcp
    - 9301:9300/tcp
        hard: -1
        soft: -1
        hard: 65536
        soft: 65536
    - /Users/pciruzzi/Documents/elasticsearch/data:/usr/share/elasticsearch/data:rw

My versions are:

$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose --version
docker-compose version 1.22.0, build f46880f

I'm working on a MacBook Pro with MacOS Sierra 10.13.6

And here is the full error:

elasticsearch    | [2018-10-18T17:32:08,613][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] using [1] data paths, mounts [[/usr/share/elasticsearch/data (osxfs)]], net usable_space [51gb], net total_space [233.5gb], spins? [possibly], types [fuse.osxfs]
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] heap size [495.3mb], compressed ordinary object pointers [true]
elasticsearch    | [2018-10-18T17:32:08,958][INFO ][o.e.n.Node               ] node name [vihZtyy] derived from node ID [vihZtyysS7eBK9E2w6g6WQ]; set [] to override
elasticsearch    | [2018-10-18T17:32:08,960][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.9.93-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elasticsearch    | [2018-10-18T17:32:08,961][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true,, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
elasticsearch    | [2018-10-18T17:32:16,895][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
elasticsearch    | [2018-10-18T17:32:16,944][INFO ][o.e.d.DiscoveryModule    ] [vihZtyy] using discovery type [zen]
elasticsearch    | [2018-10-18T17:32:18,448][INFO ][o.e.n.Node               ] initialized
elasticsearch    | [2018-10-18T17:32:18,449][INFO ][o.e.n.Node               ] [vihZtyy] starting ...
elasticsearch    | [2018-10-18T17:32:18,948][INFO ][o.e.t.TransportService   ] [vihZtyy] publish_address {}, bound_addresses {}
elasticsearch    | [2018-10-18T17:32:18,997][INFO ][o.e.b.BootstrapChecks    ] [vihZtyy] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch    | [2018-10-18T17:32:22,132][INFO ][o.e.c.s.ClusterService   ] [vihZtyy] new_master {vihZtyy}{vihZtyysS7eBK9E2w6g6WQ}{4ahFCl3ITfSw_Yn8C3dEFQ}{}{}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.h.n.Netty4HttpServerTransport] [vihZtyy] publish_address {}, bound_addresses {}
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.n.Node               ] [vihZtyy] started
elasticsearch    | [2018-10-18T17:32:22,313][INFO ][o.e.g.GatewayService     ] [vihZtyy] recovered [0] indices into cluster_state
elasticsearch    | [2018-10-18T17:32:23,423][INFO ][o.e.l.LicenseService     ] [vihZtyy] license [8c477263-7114-469f-9aa4-53616600ada0] mode [trial] - valid
elasticsearch    | [2018-10-18T17:32:28,664][INFO ][o.e.c.m.MetaDataCreateIndexService] [vihZtyy] [.monitoring-es-6-2018.10.18] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
elasticsearch    | [2018-10-18T17:32:49,129][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
elasticsearch    | java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
elasticsearch    | 	at org.apache.lucene.codecs.CodecUtil.checkHeader( ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
elasticsearch    | 	at ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
elasticsearch    | 	at org.elasticsearch.index.engine.InternalEngine.refresh( ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.shard.IndexShard.refresh( ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.maybeRefreshEngine( ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.access$400( ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal( ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$ ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.common.util.concurrent.ThreadContext$ ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker( ~[?:1.8.0_141]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor$ ~[?:1.8.0_141]
elasticsearch    | 	at [?:1.8.0_141]
elasticsearch exited with code 128

Any suggestion on why this is happening?

Thanks in advance

(Jay Modi) #2

Another user reported this same error was caused by a failing disk on their machine.

(Pablo Ciruzzi) #3

Thanks for your reply!

I'm not facing the same problem I think. When I start the image by itself with Docker, it seems to be working, but when doing it inside a Docker compose, it throws that error after some random time. Do you have any idea on that?

Thanks in advance!

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.