All request sent to the same host while init RestHighLevelClient with domain

[DJ_Zhu]

I have an es cluster with multiple node. Recently I found that the bulk requests to the cluster seems uneven.

Here is the demo code:

import org.apache.http.HttpHost;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class MainTest {

    public static void main(String[] args) throws IOException {
        // 1. initialize with domain, all the requests will only be sent to the same ip
        RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost("djzhu-test-es.info", 9201, "http")));

        // 2. initialize with ip list, it works well, the requests will be sent to different ips
//        RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(
//                new HttpHost("10.116.9.117", 9201, "http"),
//                new HttpHost("10.116.24.208", 9201, "http"),
//                new HttpHost("10.116.9.116", 9201, "http"),
//                new HttpHost("10.116.24.240", 9201, "http"),
//                new HttpHost("10.116.24.41", 9201, "http")
//        ));

        // loop to send request and monitor the loadBalance
        while (true) {
            BulkRequest bulkRequest = wrapBulkRequest(100);
            client.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ActionListener<BulkResponse>() {
                @Override
                public void onResponse(BulkResponse bulkItemResponses) {
                    int status = bulkItemResponses.status().getStatus();
                    assert status == 200;
                }

                @Override
                public void onFailure(Exception e) {
                    System.out.println("onFailure");
                }
            });
        }
    }

    /**
     * construct bulk request
     */
    private static BulkRequest wrapBulkRequest(int size) {
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.timeout(TimeValue.timeValueSeconds(60));
        bulkRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.NONE);

        for (int i = 0; i < size; i++) {
            Map<String, Object> objectMap = new HashMap<>();
            IndexRequest indexRequest = new IndexRequest()
                    .index("djzhu_test_20221222_08")
                    .type("2021")
                    .create(false)
                    .source(objectMap);
            bulkRequest.add(indexRequest);
        }
        return bulkRequest;
    }
}

Just like the code, If I init the client instance with domain (backend by our DNS service, which will update periodically (per minutes)), then all the request will go to same host -- I have noticed this by iftop -f 'port 9201' and seems the tcp traffic will only be sent to a single ip.
It is beyond my expectation, I wish the request will be sent to different nodes by the domain. Please anyone can help me.

Thanks.

[swallez (Sylvain Wallez)]

This is to be expected. When a host name has several associated IP addresses, the http client library will pick the first address to which it can successfully connect.

To achieve what you want, you'll have to resolve the name to a list of IP addresses using java.net.InetAddress.getAllByName("djzhu-test-es.info") and create a list of HttpHost from there.

[DJ_Zhu]

Create client by ip list is a way to go.

I have also found anothor method to meet our need -- Sniffer. It does not depends on DNS to return multiple ip records and will update the ip list periodically.