Java Client API to check the size of bulk queue

Elastic Stack Elasticsearch
1

Hi,
I am getting EsRejectedExecutionException after sending the logs to the bulkprocessor.

I know the queue size defaults to 200. Is there any Java API with which I can check the current queue size and handle the addition of requests accordingly?

2

Not sure but I definitely prefer using the BulkProcessor class instead of doing manual stuff... Not sure this helps though.

3

I am using bulkprocessor class. The problem is that we need to control the rate with which we are sending requests to bulkprocessor because queue is getting filled. So I want to know if there is a way get the current queue size like how we get using the nodestats call over http.

4

I see. I wish we can enhance at some point the bulk processor to slow down automatically when the cluster gets overloaded. In the same way beats and logstash are doing I think.

Anyway, I don't there is a method to access nodes stats API (I did not find it).
So here is may be a way to get that information. Note that you will have to parse the JSON response with whatever framework (Jackson?) to access the data you want to look for.

Response response = client.getLowLevelClient().performRequest(new Request("GET", "/_nodes/stats/thread_pool"));
String s = EntityUtils.toString(response.getEntity());
System.out.println("thread_pool = " + s);
5

I have one more doubt regarding this.

By default bulkprocessor
sets backoffPolicy to an exponential backoff with 8 retries and a start delay of 50ms. The total wait time is roughly 5.1 seconds.

So in node _nodes/stats/thread_pool, what happens to the rejected requests in rejected section? Are the retried again and does this number changes if the retried request succeeds?

6

I don't think this number will change.

7 
NodesStatsResponse nodesStatsResponse = client.admin().cluster().nodesStats(new NodesStatsRequest().all()).actionGet();
	for (NodeStats node : nodesStatsResponse.getNodes()) {
        if (node.getHostname().equalsIgnoreCase("NODENAME")) {
            ThreadPoolStats thread = node.getThreadPool();
            Iterator<ThreadPoolStats.Stats> itr = thread.iterator();
            while (itr.hasNext()) {
                ThreadPoolStats.Stats row = itr.next();
                if (row.getName().equalsIgnoreCase("bulk")) {
                    int currentQueueSize = row.getQueue();
                }
            }
        }

    }

I used NodesStatsResponse api to get the bulk queue size.

Please respond if this can be used to get the bulk queue size for a particular node

8

Note that you are using a TransportClient which is removed in 8.0.

The Nodes Stats API allows you to set a specific node with:

GET /_nodes/nodeId/stats/thread_pool
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.