LEAK: ByteBuf.release() was not called before it's garbage-collected

Orwell-coder · September 5, 2022, 8:11am

Hello, everyone.
I have a problem I have never encountered, so I need some help.
I have a single-type es node , the es node is runing by docker contianer.
Host Version: CentOS 8.3 x64.
Docker Version: 20.10.5.
ES Version: elasticsearch 8.3.3 or 8.4.1

jvm.options

-Xms31G
-Xmx31G
-XX:+UseG1GC

-Djava.io.tmpdir=${ES_TMPDIR}


-XX:+HeapDumpOnOutOfMemoryError

-XX:+ExitOnOutOfMemoryError

-XX:HeapDumpPath=data

-XX:ErrorFile=logs/hs_err_pid%p.log

-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

elasticsearch.yaml

network.host: 127.0.0.1
http.port: 9200
bootstrap.memory_lock: true
action.auto_create_index: true
action.destructive_requires_name: true
thread_pool.write.queue_size: 4000
xpack.security.enabled: false

[2022-09-05T07:49:19,223][ERROR][i.n.u.ResourceLeakDetector] [thp_elasticsearch] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer@4.1.77.Final/io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:386)
	io.netty.buffer@4.1.77.Final/io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:169)
	org.elasticsearch.transport.netty4@8.4.1/org.elasticsearch.transport.netty4.NettyAllocator$NoDirectBuffers.heapBuffer(NettyAllocator.java:289)
	org.elasticsearch.transport.netty4@8.4.1/org.elasticsearch.transport.netty4.NettyAllocator$1.obtain(NettyAllocator.java:137)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.ensureCapacityFromPosition(RecyclerBytesStreamOutput.java:232)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.ensureCapacity(RecyclerBytesStreamOutput.java:223)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.writeBytes(RecyclerBytesStreamOutput.java:77)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.Streams$FlushOnCloseOutputStream.writeBytes(Streams.java:236)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:504)
	com.fasterxml.jackson.core@2.13.2/com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2171)
	com.fasterxml.jackson.core@2.13.2/com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1214)
	org.elasticsearch.xcontent.impl@8.4.1/org.elasticsearch.xcontent.provider.json.JsonXContentGenerator.close(JsonXContentGenerator.java:581)
	org.elasticsearch.xcontent@8.4.1/org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1235)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.bytes.BytesReference.bytes(BytesReference.java:36)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:54)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:46)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:21)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:27)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)
	org.elasticsearch.server@8.4.1/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)
	org.elasticsearch.server@8.4.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160)
	org.elasticsearch.server@8.4.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:192)
	org.elasticsearch.server@8.4.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:186)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:612)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:593)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:582)
	org.elasticsearch.server@8.4.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160)
	org.elasticsearch.server@8.4.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:192)
	org.elasticsearch.server@8.4.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:186)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1066)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:978)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:969)
	org.elasticsearch.server@8.4.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1380)
	org.elasticsearch.server@8.4.1/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1465)
	org.elasticsearch.server@8.4.1/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1445)
	org.elasticsearch.server@8.4.1/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:39)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$3(TransportReplicationAction.java:494)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:453)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:439)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation$1.lambda$onResponse$0(ReplicationOperation.java:162)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation.updateCheckPoints(ReplicationOperation.java:340)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:158)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:153)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:301)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:443)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$1(TransportWriteAction.java:472)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.lambda$preserveContext$0(AsyncIOProcessor.java:122)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:111)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:89)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:73)
	org.elasticsearch.server@8.4.1/org.elasticsearch.index.shard.IndexShard.sync(IndexShard.java:3620)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:470)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:308)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:153)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:473)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:241)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:203)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:253)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:133)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:72)
	org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:211)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)
	org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	java.base/java.lang.Thread.run(Thread.java:833)

but I'm still confused. I didn't call netty's code directly. What should I do?
Help me please!!!!!
Thanks a lot!

I have anthor question about OOM .the elasticsearch version is 8.3.3 and running on the host is working normally.
However, when running in the docker container for a long time, the memory in the old Egen area is not released.
I don't kown what i need do it . help me please.

warkolm · September 5, 2022, 8:12am

Welcome to our community!

It's not clear what your problem is sorry, we need more information.

DavidTurner · September 5, 2022, 4:17pm

I think this is a bug in ES, thanks for reporting it. I opened RestResponse buffer leak in versions ≥8.3.1 · Issue #89808 · elastic/elasticsearch · GitHub. I haven't been able to reproduce it and don't see an obvious cause yet but we'll investigate further. I doubt there's much you can do to fix it yourself. Does it happen often? Can you make it happen on demand?

Orwell-coder · September 6, 2022, 6:29am

I am apologize , I don't know what happened to es . When I running ES in docker container this error message at show shell terminal . I just running ES in docker container by docker-compose.
docker-compose.yml

version: "3.9"
services:
  elasticsearch:
    container_name: thp-elasticsearch-test
    image: elasticsearch:8.4.1
    restart: always 
    network_mode: host
    ulimits: 
      nproc: 65535
      nofile: 
        soft: 65536
        hard: 65536
      memlock: 
        soft: -1
        hard: -1
    environment:
      - "discovery.type=single-node"
    volumes:
      - /opt/thp/config/bd/elasticsearch:/usr/share/elasticsearch/config
      - /opt/thp/data/bd/es/data:/usr/share/elasticsearch/data
      - /opt/thp/data/bd/es/logs:/usr/share/elasticsearch/logs
      - /opt/bd/etc/elasticsearch/GeoLite2-ASN.mmdb:/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb
      - /opt/bd/etc/elasticsearch/GeoLite2-City.mmdb:/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb
      - /opt/bd/etc/elasticsearch/GeoLite2-Country.mmdb:/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-Country.mmdb
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

DavidTurner · September 6, 2022, 8:26am

Ok no worries, we'll try and work it out.

Armin_Braun · September 8, 2022, 3:27pm

@Orwell-coder are you by any chance seeing any logs from org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler as well? If so could you paste them here please?
Thanks!

Orwell-coder · September 9, 2022, 12:44am

I didn't see any errors like org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler.
This is my elasticsearch running logs elasticsearch.log

system · October 7, 2022, 12:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.