Is there any way to enable versioning or concurrency control of documents based on a particular field? (Timestamp field)

Elastic Stack Elasticsearch
1

Hello,

I am currently working with catalog data that is coming in hourly in our pipeline. This is an example of how the written source document might look like.

{
    "name": "Hoodie",
    "sku": "123",
    "timeStamp": "2020-01-16T17:39:23Z",
    "price": 20
}

Lets say there is an issue and old data happens to come in later than the new data for whatever reason. Is there any way to tell ElasticSearch to look at the timeStamp field first before writing the document?

For example, lets say the document above was written, and a few mins later this document comes in...

{
    "name": "Hoodie",
    "sku": "123",
    "timeStamp": "2019-12-31T17:39:23Z",
    "price": 15
}

I obviously wouldn't want this to overwrite the more recent one. Is there a way to let ES know to version the documents by a particular field? I looked into using a script, but i can't figure out a way to have access to both the _source context and the incoming document context.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.