Data loss- injection using transport client

[hen_kling]

Hi,
I sent bluk to elasticsearch via Nifi using PutElasticsearch5 processor.

In the Nifi processor code:
When sent to elastic sesrch using transport client, i got an array of responses for each in the bulk.
I checked if there was a problem, using elastic client libraries:
Org.elasticsearch.action.BulkItemResponse
Org.elssticsearch.action.BulkResponse
For every bulk we got a successful response ( BulkResponce class) for each doc in bulk.

Then, i encountered a problem:
When accessed to the index in ES, i noticed that there was some missing docs (about 0.7%)
I checked the logs in the ES environment and didn't see anything that indicates loss of data in the index.
The elastic env was stable during the insert, and also the Nifi environment.

Any idea what could cause this situation?
Thank you

[DavidTurner]

@hen_kling

For every bulk we got a successful response ( BulkResponce class) for each doc in bulk.

This doesn't quite make sense, and the ambiguity is important to resolve since it is germane to your question. You get one BulkResponse for each bulk request, and it contains many BulkItemResponse objects. Each BulkItemResponse might be successful or failed, which you can check with BulkResponse#hasFailures(). What exactly do you mean here?

Also, did you remember to refresh the index before your searches?