Elasticsearch 6.1.1 client running OOM and dumping heap

cching · October 1, 2019, 8:45pm

We are having troubles with Elasticsearch 6.1.1 running in client mode. Basically it runs as a frontend for bulk requests from log stash and queries from Kibana. Analyzing the heap dump shows about half the memory is taken by the clientHandlers ConcurrentMapLong in Transport:

github.com

elastic/elasticsearch/blob/6.1/core/src/main/java/org/elasticsearch/transport/TransportService.java#L91


protected final ThreadPool threadPool;
protected final ClusterName clusterName;
protected final TaskManager taskManager;
private final TransportInterceptor.AsyncSender asyncSender;
private final Function<BoundTransportAddress, DiscoveryNode> localNodeFactory;
private final boolean connectToRemoteCluster;


volatile Map<String, RequestHandlerRegistry> requestHandlers = Collections.emptyMap();
final Object requestHandlerMutex = new Object();


final ConcurrentMapLong<RequestHolder> clientHandlers = ConcurrentCollections.newConcurrentMapLongWithAggressiveConcurrency();


final CopyOnWriteArrayList<TransportConnectionListener> connectionListeners = new CopyOnWriteArrayList<>();


private final TransportInterceptor interceptor;


// An LRU (don't really care about concurrency here) that holds the latest timed out requests so if they
// do show up, we can print more descriptive information about them
final Map<Long, TimeoutInfoHolder> timeoutInfoHandlers =
    Collections.synchronizedMap(new LinkedHashMap<Long, TimeoutInfoHolder>(100, .75F, true) {
        @Override

Can anyone provide some insight into how we can prevent this? Is it possibly a memory leak? Happy to provide more information!

system · October 29, 2019, 8:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.