Preventing document field from getting updated in Elastic Search

Elastic Stack Elasticsearch
1

I have the following requirement. In the system there is a job running that fetches the documents from another system every 15 mins. Once those documents are fetched, I have my own job which runs and updates the document and inserts some fields into it. Like the document which was fetched from another system is something like this.

{
  "id": 1,
  "firstName": "XYZ",
  "lastName": "ABC",
  "grade": "",
  "isUpdaterequired":"true",
  "details":{
    "firstName": null,
    "lastName": null,
    "grade": null,
    "otherDetails":null
  }
}

Now once this document is ingested then we run our own job and modifies the document like this:

{
  "id": 1,
  "firstName": "XYZ",
  "lastName": "ABC",
  "grade": "",
  "isUpdaterequired":false,
  "details":{
    "firstName": "XYZ",
    "lastName": "ABC",
    "grade": "1.10",
    "otherDetails":"Filled from some other source"
  }
}

Now after the 15 mins data fetch job runs again it gets the documents and the same document comes again

{
  "id": 1,
  "firstName": "XYZ",
  "lastName": "ABC",
  "grade": "",
  "isUpdaterequired":"true",
  "details":{
    "firstName": null,
    "lastName": null,
    "grade": null,
    "otherDetails":null
  }
}

So my job will save the document again and the "details" filed will be replaced. Can we prevent this specific field from getting updated using the "isUpdaterequired" field or is there any other way to stop updating a field in the time of upsert?

I am using an Elastic search with Spring boot.

Querying the presence of the document before every save is expensive as we received almost 10k documents at every 15 mins.

Thank you!!

2

That is something you will have to control from your application code. Elasticsearch has no way to enforce restrictions at the field level within a document.

3

For writes, yes.
There is read level controls :slight_smile:

4

Could you please give me some details about it or things to look in the elastic docs. That would be a great help.

5

https://www.elastic.co/guide/en/elasticsearch/reference/current/field-level-security.html

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.