Frequent Creation of Threads by elastic.clients

Elastic Stack Elasticsearch
1

I am using Elasticsearch Java API Client [8.13] and Elasticsearch (8.13), but I encounter the following exception:

复制代码

unable to create native thread: possibly out of memory or process/resource limits reached
    at java.base/java.lang.Thread.start0(Native Method)
    at java.base/java.lang.Thread.start(Thread.java:809)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.start(CloseableHttpAsyncClientBase.java:83)
    at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:302)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao.<init>(FixQueryTeacherByEsDao.java:75)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao$FastClassByGuice$49598721.GUICE$TRAMPOLINE(<generated>)
    at kepler.schedulesearcher.dao.FixQueryTeacherByEsDao$FastClassByGuice$49598721.apply(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1100)
     ... 38 common frames omitted

I use VisualVM to monitor my project, and I found many threads similar to elasticsearch-rest-client-550-thread-2.

When I restart the project, I notice that every request adds two groups of similar threads, each group having 9 threads.

I have tried creating a singleton ElasticSearchClient object in the project to share this object, but every time I use this object, such threads are still created.

Occasionally, in concurrent situations, CPU usage reaches 100%. I am not sure if the memory overflow is related to this phenomenon, but I want to confirm whether it is normal for the ElasticSearch-rest-client-()- thread-() threads to be created frequently.

my code：

@Singleton
public class ElasticsearchClientManagerUtils {
    final static Logger logger
            = LoggerFactory.getLogger(ElasticSearchConnectUtils.class);
    private static ElasticsearchClientManagerUtils instance;
    private static RestClient restClient;
    private static ElasticsearchClient client;
    private static ElasticsearchTransport transport;
    private ElasticsearchClientManagerUtils() {
        try {
            String elasticSearchHost ="xxx.xxx.xxx.xxx";
            String elasticSearchPort = "9200";
            String elasticSearchSchema = "https";
            String elasticSearchUser = "elastic";
            String elasticSearchPassword = "**************";
            String caCertificate = "*********************";
            String credentials = elasticSearchUser + ":" + elasticSearchPassword;
            String esAuthorization = Base64.getEncoder().encodeToString(credentials.getBytes());
            RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
                return httpAsyncClientBuilder;
            });
            Header[] defaultHeaders =
                    new Header[]{new BasicHeader("Authorization",
                            "Basic "+esAuthorization)};
            SSLContext sslContext = TransportUtils
                    .sslContextFromCaFingerprint(caCertificate);
            BasicCredentialsProvider credsProv = new BasicCredentialsProvider();
            credsProv.setCredentials(
                    AuthScope.ANY, new UsernamePasswordCredentials(elasticSearchUser, elasticSearchPassword)
            );
            builder.setDefaultHeaders(defaultHeaders);
            builder.setHttpClientConfigCallback(hc->hc.setSSLContext(sslContext).setDefaultCredentialsProvider(credsProv));
            restClient = builder.build();
            // Create the transport with a Jackson mapper
            transport = new RestClientTransport(
                    restClient, new JacksonJsonpMapper());
            client = new ElasticsearchClient(transport);
        } catch (Exception e) {
            logger.info("create esClient error {}",e.getMessage());
        }
    }
    public static synchronized ElasticsearchClientManagerUtils getInstance() {
        if (instance == null) {
            instance = new ElasticsearchClientManagerUtils();
        }
        logger.info(instance.toString());
        return instance;
    }


    public ElasticsearchClient getClient() {
        return client;
    }


    public void close() {
        try {
            if (restClient != null) {
                restClient.close();
                transport.close();
            }
        } catch (Exception e) {
            logger.info("close es client error:{}",e.getMessage());
        }
    }
}

Monitoring content:

Snipaste_10-49-22-217-217
Snipaste_10-49-22-217-2171131×1034 277 KB

2

From Elastic Search to Elasticsearch

3

Added language-clients

4

This name indicates that you have created 550 separate REST client instances. You should have only one.

This stack trace indicates you're calling RestClientBuilder#build to create a new client instance each time you construct a new FixQueryTeacherByEsDao instance. That sounds very wrong to me.

3 Likes
5

@tom_ding can you apply some loggers in your private ElasticsearchClientManagerUtils() so that we are sure that for any reason this snippet is not getting triggered more than once

RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
                return httpAsyncClientBuilder;
            });

Also are you running multiple instance of your java code on K8 etc?

6

The stack trace shows that it's not that.

If so, they'd be different processes, but the OP shows there are hundreds of client instances within a single process. So it's not that either.

7

Thank you for your reply.

Yes, the stack trace information is not from my provided code.

The code I am using creates an instance for each request and then releases the resources. It looks something like this:

RestClientBuilder builder = RestClient.builder(
                new HttpHost(EsHost, Integer.parseInt(EsPort), EsScheme));
Header[] defaultHeaders =
                new Header[]{new BasicHeader("Authorization",
                        "Basic "+EsAuthorization)};
SSLContext sslContext = TransportUtils
                .sslContextFromCaFingerprint(caCertificate);
BasicCredentialsProvider credsProv = new BasicCredentialsProvider();
credsProv.setCredentials(
                AuthScope.ANY, new UsernamePasswordCredentials(EsUserName, EsUserPassword)
);
builder.setDefaultHeaders(defaultHeaders);
builder.setHttpClientConfigCallback(hc->hc.setSSLContext(sslContext).setDefaultCredentialsProvider(credsProv));
RestClient restClient = builder.build();
// Create the transport with a Jackson mapper
ElasticsearchTransport transport = new RestClientTransport(
                restClient, new JacksonJsonpMapper());
// And create the API client
ElasticsearchClient esClient = new ElasticsearchClient(transport);
....
....
....
restClient.close();

The code I initially provided was an attempt to create a singleton ElasticsearchClient object to see if it would still repeatedly create elasticsearch-rest-client-%d-thread-%d threads. The result is that it still creates a lot of threads.

8 
RestClientBuilder builder = RestClient.builder(
                    new HttpHost(elasticSearchHost, Integer.parseInt(elasticSearchPort), elasticSearchSchema)).setHttpClientConfigCallback(httpAsyncClientBuilder -> {
                httpAsyncClientBuilder.setMaxConnTotal(2);
                httpAsyncClientBuilder.setMaxConnPerRoute(2); 
logger.info("create RestClientBuilder");
                return httpAsyncClientBuilder;
            });

ElasticsearchClient client = ElasticsearchClientManagerUtils.getInstance().getClient();
logger.info("Get ElasticsearchClient "+client.toString());

I have used logging:

15:39:11.283 [qtp210281271-21] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 4.3.1.Final
15:39:11.435 [qtp210281271-21] INFO  k.s.utils.ElasticSearchConnectUtils - kepler.schedulesearcher.utils.ElasticsearchClientManagerUtils@7c2160fe
15:39:11.435 [qtp210281271-21] INFO  k.s.dao.QueryTeacherByEsDao - Get ElasticsearchClient co.elastic.clients.elasticsearch.ElasticsearchClient@11d3370a
15:39:36.517 [qtp210281271-15] INFO  k.s.utils.ElasticSearchConnectUtils - kepler.schedulesearcher.utils.ElasticsearchClientManagerUtils@7c2160fe
15:39:36.517 [qtp210281271-15] INFO  k.s.dao.QueryTeacherByEsDao - Get ElasticsearchClient co.elastic.clients.elasticsearch.ElasticsearchClient@11d3370a

more threads will still be created

9

Snipaste_15-45-58-144-144
Snipaste_15-45-58-144-1441543×567 27.5 KB

10

You're still creating loads of separate client instances. Create one, and reuse it for every request.

1 Like
11

Thank you for pointing out the cause of the problem. I will fix it in the program.

12

If you are using Spring Boot, you can leverage the ElasticsearchConfiguration class with the spring-boot-starter-data-elasticsearch library and override the clientConfiguration

13

Thanks for your response, I'm not using Spring Boot. My application did create a large number of client instances, which I have now solved.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.