Get the only failed document response in Bulk API Elasticsearch

Elastic Stack Elasticsearch
1

I am struggling with Bulk API. I am sending 100 request (index, update) in every Bulk request. It gives me response with status of each request. Suppose my 97th request got fail, I have to make it loop to find the particular error document. I think its not optimize way. If i am sending more number of Bulk request, It makes my process slow. Is there any way where i will get only failed document or count of fail/success document in response? I am using php-elasticsearch SDK .

2

There is a top-level "errors" field you can check. If that field is false, there are no errors and you don't have to loop through the documents. More details here: https://github.com/elastic/elasticsearch/issues/25156#issuecomment-307806954

E.g.

$responses = $client->bulk($params);
if (isset($responses['errors']) === true && $responses['errors'] === true) {
    echo "There was one or more errors in the bulk response.\n\n";
}
foreach ($responses['items'] as $item) {
    echo "Status: {$item['create']['status']}\n";
    if (isset($item['create']['error']) === true) {
        print_r($item['create']['error']);
    }
    echo "\n";
}
3

Its good for small bulk size what if my bulk size is much like 5000 or more it makes my process slow. I guess this is not optimize way to solve large bulk size. It will unnecessary iterate successful records. I suggest ES should enhance response format.

4

It's been discussed before (here and here) but ultimately we've decided a change isn't worth it.

It's trivial for a client to iterate over the response and the bulk response is very compressible, so even large responses have pretty low impact on network transfer speed. If your code is slow to process the bulk it may be an issue with your application code. Iterating over an array of a few thousand items should be very fast no matter the language (even in PHP).

5

Yes but I think when bulk size high it can cause speed issue.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.