ES Docker unsafe memory access


(Pablo Ciruzzi) #1

Hi,

I'm using Docker Compose to start an ES (Version 5.5.3) for development. I followed the guide to do it from here. The problem is that, after some time, I get an error:

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Here is my Docker Compose config:

$ docker-compose config
services:
  elasticsearch:
    container_name: elasticsearch
    environment:
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      bootstrap.memory_lock: "true"
      xpack.security.enabled: "false"
      xpack.watcher.enabled: "false"
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.3
    networks:
      redo_backend: null
    ports:
    - 9201:9200/tcp
    - 9301:9300/tcp
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        hard: 65536
        soft: 65536
    volumes:
    - /Users/pciruzzi/Documents/elasticsearch/data:/usr/share/elasticsearch/data:rw

My versions are:

$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose --version
docker-compose version 1.22.0, build f46880f

I'm working on a MacBook Pro with MacOS Sierra 10.13.6

And here is the full error:

elasticsearch    | [2018-10-18T17:32:08,613][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] using [1] data paths, mounts [[/usr/share/elasticsearch/data (osxfs)]], net usable_space [51gb], net total_space [233.5gb], spins? [possibly], types [fuse.osxfs]
elasticsearch    | [2018-10-18T17:32:08,930][INFO ][o.e.e.NodeEnvironment    ] [vihZtyy] heap size [495.3mb], compressed ordinary object pointers [true]
elasticsearch    | [2018-10-18T17:32:08,958][INFO ][o.e.n.Node               ] node name [vihZtyy] derived from node ID [vihZtyysS7eBK9E2w6g6WQ]; set [node.name] to override
elasticsearch    | [2018-10-18T17:32:08,960][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.9.93-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elasticsearch    | [2018-10-18T17:32:08,961][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
...
elasticsearch    | [2018-10-18T17:32:16,895][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
elasticsearch    | [2018-10-18T17:32:16,944][INFO ][o.e.d.DiscoveryModule    ] [vihZtyy] using discovery type [zen]
elasticsearch    | [2018-10-18T17:32:18,448][INFO ][o.e.n.Node               ] initialized
elasticsearch    | [2018-10-18T17:32:18,449][INFO ][o.e.n.Node               ] [vihZtyy] starting ...
elasticsearch    | [2018-10-18T17:32:18,948][INFO ][o.e.t.TransportService   ] [vihZtyy] publish_address {172.18.0.4:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch    | [2018-10-18T17:32:18,997][INFO ][o.e.b.BootstrapChecks    ] [vihZtyy] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch    | [2018-10-18T17:32:22,132][INFO ][o.e.c.s.ClusterService   ] [vihZtyy] new_master {vihZtyy}{vihZtyysS7eBK9E2w6g6WQ}{4ahFCl3ITfSw_Yn8C3dEFQ}{172.18.0.4}{172.18.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.h.n.Netty4HttpServerTransport] [vihZtyy] publish_address {172.18.0.4:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch    | [2018-10-18T17:32:22,173][INFO ][o.e.n.Node               ] [vihZtyy] started
elasticsearch    | [2018-10-18T17:32:22,313][INFO ][o.e.g.GatewayService     ] [vihZtyy] recovered [0] indices into cluster_state
...
elasticsearch    | [2018-10-18T17:32:23,423][INFO ][o.e.l.LicenseService     ] [vihZtyy] license [8c477263-7114-469f-9aa4-53616600ada0] mode [trial] - valid
elasticsearch    | [2018-10-18T17:32:28,664][INFO ][o.e.c.m.MetaDataCreateIndexService] [vihZtyy] [.monitoring-es-6-2018.10.18] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
elasticsearch    | [2018-10-18T17:32:49,129][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[vihZtyy][refresh][T#1]], exiting
elasticsearch    | java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
elasticsearch    | 	at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
...
elasticsearch    | 	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
elasticsearch    | 	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:910) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService.access$400(IndexService.java:92) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.5.3.jar:5.5.3]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141]
elasticsearch    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141]
elasticsearch    | 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
elasticsearch exited with code 128

Any suggestion on why this is happening?

Thanks in advance


(Jay Modi) #2

Another user reported this same error was caused by a failing disk on their machine. https://github.com/elastic/elasticsearch/issues/32163


(Pablo Ciruzzi) #3

Thanks for your reply!

I'm not facing the same problem I think. When I start the image by itself with Docker, it seems to be working, but when doing it inside a Docker compose, it throws that error after some random time. Do you have any idea on that?

Thanks in advance!


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.