Updating ES records : Ingestion lag and Search returns empty

Hi ES team,

We currently have a Nodejs import script running to process our XML,HTML files followed by ingestion to ES. The XML,HTML files are uploaded in S3 bucket and queued up in the SQS triggering the Lambda function containing Nodejs import script. In the import script, one record contains 2 language fields and both the language has their own set of XML,HTML files but share a unique code. So before inserting a record for one language we search the ES for the unique code and if it exists then we update that record by retrieving the doc_id else we create a new record.

We are facing an issue when the ingestion of new record and a language update comes in the queue one after the other. The first record is in process n sent to ES for ingestion and the node event loop picks up the next language record and does the search for the unique field returns empty because there is an ingestion lag so this results in 2 records getting created for each language making the defeating the purpose of the unique field. but if I try it after 2-3 seconds it works

How to handle this issue ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.