ES Docker unsafe memory access

Hi,

I'm using Docker Compose to start an ES (Version 5.5.3) for development. I followed the guide to do it from here. The problem is that, after some time, I get an error:

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Here is my Docker Compose config:

$ docker-compose config
services:
  elasticsearch:
    container_name: elasticsearch
    environment:
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      bootstrap.memory_lock: "true"
      xpack.security.enabled: "false"
      xpack.watcher.enabled: "false"
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.3
    networks:
      redo_backend: null
    ports:
    - 9201:9200/tcp
    - 9301:9300/tcp
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        hard: 65536
        soft: 65536
    volumes:
    - /Users/pciruzzi/Documents/elasticsearch/data:/usr/share/elasticsearch/data:rw

My versions are:

$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose --version
docker-compose version 1.22.0, build f46880f

I'm working on a MacBook Pro with MacOS Sierra 10.13.6

And here is the full error:

elasticsearch    | [2018-10-18T17:32:08,613][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] using [1] data paths, mounts [[/usr/share/elasticsearch/data (osxfs)]], net usable_space [51gb], net total_space [233.5gb], spins? [possibly], types [fuse.osxfs]
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] heap size [495.3mb], compressed ordinary object pointers [true]
elasticsearch    | [2018-10-18T17:32:08,958][INFO ][o.e.n.Node               ] node name [vihZtyy] derived from node ID [vihZtyysS7eBK9E2w6g6WQ]; set [node.name] to override
elasticsearch    | [2018-10-18T17:32:08,960][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.9.93-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elasticsearch    | [2018-10-18T17:32:08,961][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
...
elasticsearch    | [2018-10-18T17:32:16,895][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
elasticsearch    | [2018-10-18T17:32:16,944][INFO ][o.e.d.DiscoveryModule    ] [vihZtyy] using discovery type [zen]
elasticsearch    | [2018-10-18T17:32:18,448][INFO ][o.e.n.Node               ] initialized
elasticsearch    | [2018-10-18T17:32:18,449][INFO ][o.e.n.Node               ] [vihZtyy] starting ...
elasticsearch    | [2018-10-18T17:32:18,948][INFO ][o.e.t.TransportService   ] [vihZtyy] publish_address {172.18.0.4:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch    | [2018-10-18T17:32:18,997][INFO ][o.e.b.BootstrapChecks    ] [vihZtyy] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch    | [2018-10-18T17:32:22,132][INFO ][o.e.c.s.ClusterService   ] [vihZtyy] new_master {vihZtyy}{vihZtyysS7eBK9E2w6g6WQ}{4ahFCl3ITfSw_Yn8C3dEFQ}{172.18.0.4}{172.18.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.h.n.Netty4HttpServerTransport] [vihZtyy] publish_address {172.18.0.4:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.n.Node               ] [vihZtyy] started
elasticsearch    | [2018-10-18T17:32:22,313][INFO ][o.e.g.GatewayService     ] [vihZtyy] recovered [0] indices into cluster_state
...
elasticsearch    | [2018-10-18T17:32:23,423][INFO ][o.e.l.LicenseService     ] [vihZtyy] license [8c477263-7114-469f-9aa4-53616600ada0] mode [trial] - valid
elasticsearch    | [2018-10-18T17:32:28,664][INFO ][o.e.c.m.MetaDataCreateIndexService] [vihZtyy] [.monitoring-es-6-2018.10.18] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
elasticsearch    | [2018-10-18T17:32:49,129][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
elasticsearch    | java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
elasticsearch    | 	at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
...
elasticsearch    | 	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
elasticsearch    | 	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:910) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.access$400(IndexService.java:92) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141]
elasticsearch    | 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
elasticsearch exited with code 128

Any suggestion on why this is happening?

Thanks in advance

Another user reported this same error was caused by a failing disk on their machine. https://github.com/elastic/elasticsearch/issues/32163

Thanks for your reply!

I'm not facing the same problem I think. When I start the image by itself with Docker, it seems to be working, but when doing it inside a Docker compose, it throws that error after some random time. Do you have any idea on that?

Thanks in advance!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.