Frequent Creation of Threads by elastic.clients

I am using Elasticsearch Java API Client [8.13] and Elasticsearch (8.13), but I encounter the following exception:

复制代码

unable to create native thread: possibly out of memory or process/resource limits reached
    at java.base/java.lang.Thread.start0(Native Method)
    at java.base/java.lang.Thread.start(Thread.java:809)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.start(CloseableHttpAsyncClientBase.java:83)
    at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:302)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao.<init>(FixQueryTeacherByEsDao.java:75)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao$FastClassByGuice$49598721.GUICE$TRAMPOLINE(<generated>)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao$FastClassByGuice$49598721.apply(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1100)
     ... 38 common frames omitted

I use VisualVM to monitor my project, and I found many threads similar to elasticsearch-rest-client-550-thread-2.

When I restart the project, I notice that every request adds two groups of similar threads, each group having 9 threads.

I have tried creating a singleton ElasticSearchClient object in the project to share this object, but every time I use this object, such threads are still created.

Occasionally, in concurrent situations, CPU usage reaches 100%. I am not sure if the memory overflow is related to this phenomenon, but I want to confirm whether it is normal for the ElasticSearch-rest-client-()- thread-() threads to be created frequently.

my code:

@Singleton
public class ElasticsearchClientManagerUtils {
    final static Logger logger
            = LoggerFactory.getLogger(ElasticSearchConnectUtils.class);
    private static ElasticsearchClientManagerUtils instance;
    private static RestClient restClient;
    private static ElasticsearchClient client;
    private static ElasticsearchTransport transport;
    private ElasticsearchClientManagerUtils() {
        try {
            String elasticSearchHost ="xxx.xxx.xxx.xxx";
            String elasticSearchPort = "9200";
            String elasticSearchSchema = "https";
            String elasticSearchUser = "elastic";
            String elasticSearchPassword = "**************";
            String caCertificate = "*********************";
            String credentials = elasticSearchUser + ":" + elasticSearchPassword;
            String esAuthorization = Base64.getEncoder().encodeToString(credentials.getBytes());
            RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
                return httpAsyncClientBuilder;
            });
            Header[] defaultHeaders =
                    new Header[]{new BasicHeader("Authorization",
                            "Basic "+esAuthorization)};
            SSLContext sslContext = TransportUtils
                    .sslContextFromCaFingerprint(caCertificate);
            BasicCredentialsProvider credsProv = new BasicCredentialsProvider();
            credsProv.setCredentials(
                    AuthScope.ANY, new UsernamePasswordCredentials(elasticSearchUser, elasticSearchPassword)
            );
            builder.setDefaultHeaders(defaultHeaders);
            builder.setHttpClientConfigCallback(hc->hc.setSSLContext(sslContext).setDefaultCredentialsProvider(credsProv));
            restClient = builder.build();
            // Create the transport with a Jackson mapper
            transport = new RestClientTransport(
                    restClient, new JacksonJsonpMapper());
            client = new ElasticsearchClient(transport);
        } catch (Exception e) {
            logger.info("create esClient error {}",e.getMessage());
        }
    }
    public static synchronized ElasticsearchClientManagerUtils getInstance() {
        if (instance == null) {
            instance = new ElasticsearchClientManagerUtils();
        }
        logger.info(instance.toString());
        return instance;
    }


    public ElasticsearchClient getClient() {
        return client;
    }


    public void close() {
        try {
            if (restClient != null) {
                restClient.close();
                transport.close();
            }
        } catch (Exception e) {
            logger.info("close es client error:{}",e.getMessage());
        }
    }
}

Monitoring content:

From Elastic Search to Elasticsearch

Added language-clients

This name indicates that you have created 550 separate REST client instances. You should have only one.

This stack trace indicates you're calling RestClientBuilder#build to create a new client instance each time you construct a new FixQueryTeacherByEsDao instance. That sounds very wrong to me.

3 Likes

@tom_ding can you apply some loggers in your private ElasticsearchClientManagerUtils() so that we are sure that for any reason this snippet is not getting triggered more than once

RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
                return httpAsyncClientBuilder;
            });

Also are you running multiple instance of your java code on K8 etc?

The stack trace shows that it's not that.

If so, they'd be different processes, but the OP shows there are hundreds of client instances within a single process. So it's not that either.

Thank you for your reply.

Yes, the stack trace information is not from my provided code.

The code I am using creates an instance for each request and then releases the resources. It looks something like this:

RestClientBuilder builder = RestClient.builder(
                new HttpHost(EsHost, Integer.parseInt(EsPort), EsScheme));
Header[] defaultHeaders =
                new Header[]{new BasicHeader("Authorization",
                        "Basic "+EsAuthorization)};
SSLContext sslContext = TransportUtils
                .sslContextFromCaFingerprint(caCertificate);
BasicCredentialsProvider credsProv = new BasicCredentialsProvider();
credsProv.setCredentials(
                AuthScope.ANY, new UsernamePasswordCredentials(EsUserName, EsUserPassword)
);
builder.setDefaultHeaders(defaultHeaders);
builder.setHttpClientConfigCallback(hc->hc.setSSLContext(sslContext).setDefaultCredentialsProvider(credsProv));
RestClient restClient = builder.build();
// Create the transport with a Jackson mapper
ElasticsearchTransport transport = new RestClientTransport(
                restClient, new JacksonJsonpMapper());
// And create the API client
ElasticsearchClient esClient = new ElasticsearchClient(transport);
....
....
....
restClient.close();

The code I initially provided was an attempt to create a singleton ElasticsearchClient object to see if it would still repeatedly create elasticsearch-rest-client-%d-thread-%d threads. The result is that it still creates a lot of threads.

RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
logger.info("create RestClientBuilder");
                return httpAsyncClientBuilder;
            });

ElasticsearchClient client = ElasticsearchClientManagerUtils.getInstance().getClient();
logger.info("Get ElasticsearchClient "+client.toString());

I have used logging:

15:39:11.283 [qtp210281271-21] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 4.3.1.Final
15:39:11.435 [qtp210281271-21] INFO  k.s.utils.ElasticSearchConnectUtils - kepler.schedulesearcher.utils.ElasticsearchClientManagerUtils@7c2160fe
15:39:11.435 [qtp210281271-21] INFO  k.s.dao.QueryTeacherByEsDao - Get ElasticsearchClient co.elastic.clients.elasticsearch.ElasticsearchClient@11d3370a
15:39:36.517 [qtp210281271-15] INFO  k.s.utils.ElasticSearchConnectUtils - kepler.schedulesearcher.utils.ElasticsearchClientManagerUtils@7c2160fe
15:39:36.517 [qtp210281271-15] INFO  k.s.dao.QueryTeacherByEsDao - Get ElasticsearchClient co.elastic.clients.elasticsearch.ElasticsearchClient@11d3370a

more threads will still be created

You're still creating loads of separate client instances. Create one, and reuse it for every request.

Thank you for pointing out the cause of the problem. I will fix it in the program.

If you are using Spring Boot, you can leverage the ElasticsearchConfiguration class with the spring-boot-starter-data-elasticsearch library and override the clientConfiguration

Thanks for your response, I'm not using Spring Boot. My application did create a large number of client instances, which I have now solved.