Partial bulk insert into elasticsearch using rest API

Elastic Stack Elasticsearch
1

Hi All,
I am trying to bulk insert data into elastic search batch wise(say 10 records at once). Out of these I am having one record which is invalid. The rest client is rolling back all the 10 records. I want to save the valid 9 records and fail only the 1 records. Can someone please help me to figure out a way to do this?

Regards
Santosh

2

There is no rollback within a bulk request so what makes you think the rest client rolls back?

4

hi
I have another setup with the older version 6.x, using transport client, that actually saves the valid records within a batch of bulk insert, even if there is a failure record.
Whereas the similar code using 7.x with rest client does not do so, it doesn't save even the valid records. it rejects all the rows.

thanks and regards
Santosh

5

If it rejects all the rows it means that none of them are valid or that there is something wrong with your request. Check the bulk response for error messages.

6

hi
I am intentionally giving one row invalid out of 10, if i correct that row all the 10 records get inserted.
I want elastic search to insert the 9 valid rows, but it rejects all the rows.
The same data when I try in 6.x version with transport code it inserts the 9 records and the 1 invalid record is rejected.

thanks and regards
Santosh

7

What error is returned for the 9 records? What does the bulk request look like? Are you maybe using a document type which has been deprecated in ES 7.x? Are you maybe having mapping issues due to changes in ES 7.x?

What error message do you get if you index a single document wothout using bulk?

1 Like
8

hi
thanks for the help. Got the problem. There are two things:

  1. Field level validations, like improperly formed JSON field and so on.
  2. duplicate data and other validations that happen once the data reaches elastic search server.
    When doing batch processing, error of type one make the whole batch rollback or fail. Where as error type two persist valid data into elastic search server and elastic search server will give proper error messages to us.
    My issue is solved :slight_smile:
    Thanks and regards
    Santosh
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.