I have been using bulk processor API to send messages to elastic search.Some times elastic gets restarted often, then i observed that I lost some messages.Is this because of elastic or bulk processor.
Please suggest me.
Bulkprocessor does not retry in case of failure.
You must read the bulk response to see if there are individual failures or catch the Exception in case of a global failure.
Then retry if needed.
But stopping elasticsearch often is not a best practice. Why are you doing that?
Hi David,
Thanks for your reply.
Some times Even if i don't retry also i can able to see messages sent to elastic search once it is started.Some times i have observed that adding failed messages leading to duplicate sending to elastic search.
Suppose i am sending 100000 records to elastic search using bulk processor, If elastic search gets down while processing bulkprocessor requests.@afterbulk method called with throwable exception and failed request.
If add failed messages to bulk processor and restarted elastic then observed that 100100 messages received by elastic search .
If I did not add failed messages then observed that 99900 messages received by elastic search .
Why is this happening?
Suggest me if i am wrong.
If elasticsearch goes down before it proceeds the full bulk, then nothing should be indexed.
If you get a response from elasticsearch, it means it is at least in the transaction log. So when your node restart it should be played again.
But do you have only one node?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.