Nest BulkAll - Error: 'BulkAll halted after receiving failures that can not be retried from _bulk'

Elastic Stack Elasticsearch
,
1

I am currently using a foreach to index through a pipeline using the nest client. This is slow and highly inefficient. But works.

For that same dataset, I tried using the BulkAll command as follows to push the same data through the same pipeline during indexing. However, this results in errors.

Error: "BulkAll halted after receiving failures that can not be retried from _bulk"

Looking at the reason it says its a 201 status code which to me means created. And I see data showing up in the index as well.

Here is the code I am using to push the data into the index through the pipeline.

    public void IngestViaPipeline<T>(IEnumerable<T> data, string indexName, string pipelineName) where T : class {
        var pageSize = 1;
        var bulkAllObservable = _client.BulkAll(data, b => b
            .Index(indexName)
            .Pipeline(pipelineName)
            .BackOffRetries(10)
            .BackOffTime("30s")
            .RefreshOnCompleted()
            .MaxDegreeOfParallelism(Environment.ProcessorCount)
            .Size(pageSize)
            .RetryDocumentPredicate((item, doc) => { return true; })
            .DroppedDocumentCallback((item, doc) => {
                _logger.LogError($"Could not index doc.{Environment.NewLine}{item}{Environment.NewLine}{System.Text.Json.JsonSerializer.Serialize(doc)}");
            })
        );

        var waitHandle = new ManualResetEvent(false);
        ExceptionDispatchInfo exceptionDispatchInfo = null;

        var observer = new BulkAllObserver(
            onNext: response => {
                _logger.LogDebug($"Bulk Indexing Page: {response.Page} out of {data.Count() / pageSize} for Index: {indexName} using Pipeline: {pipelineName}");
            },
            onError: exception => {
                exceptionDispatchInfo = ExceptionDispatchInfo.Capture(exception);
                waitHandle.Set();
            },
            onCompleted: () => waitHandle.Set()
        );

        bulkAllObservable.Subscribe(observer);
        waitHandle.WaitOne();
        exceptionDispatchInfo?.Throw();
    }

I have looked at the code long enough and tried about everything I can find in the docs about this command. So I am sure it is fatigue letting me miss the critical thing. What can I do differently to pinpoint the cause? Is there anything I am doing incorrectly here?

2

Hi @dumstattd. I think you're missing .ContinueAfterDroppedDocuments() in your BulkAll configuration which enable the continuation after any failed documents. Those failures should then invoke the callback.

1 Like
3

Hello @stevejgordon That did not seem to do the trick. After that I did not get anymore errors reported than before.

Error shown:

MyService[0]
      Elasticsearch.Net.ElasticsearchClientException: Bulk indexing failed and after retrying 10 times
         at Nest.BulkAllObservable`1.BulkAsync(IList`1 buffer, Int64 page, Int32 backOffRetries)
         at Nest.BulkAllObservable`1.RetryDocuments(Int64 page, Int32 backOffRetries, IList`1 retryDocuments)

I did add in more logging within RetryDocumentPredicate in order to show the item and the doc value. The Item is as such.

Retrying to index doc.
      index returned 200 _index: myindex _type:  _id: 1 _version: 14 error:

The data itself I can send using IndexManyAsync and it does not have a problem with it. But when sending through BulkAll it seems to not accept the document.

4

Does the json serializer change between bulk and IndexManyAsync? I can take that exact document and post it to the index using the pipeline through VSCode Elastic Client and it will only work when using camel case. Does bulk not send via camel case?

5

I just flipped it to use Index instead of bulkall. With a foreach wrapping it and it works just fine using the same exact data. So then I flipped over to using Bulk instead of BulkAll and put in my own paging method to page through 1000 record chunks. Again works flawlessly. So too me it starts to either point to a bug in BulkAll or that the documentation is incorrect/incomplete.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

7

@dumstattd I think this is indeed a bug and I think I've found the cause.

Created the following issue to explain and track the failure.