Best way to update a single field across all the documents (may be million docs)

Elastic Stack Elasticsearch
1

Team,

Can I get some suggestions around updating a single field across multiple documents in the elasticsearch? Thing here is need to get a value of that field (existing) compare it will an external value and do some calculations and replace it in place update. Was thinking to use updatebyquery because of scroll performance but need help in how best we can pass through the external data value.. or is there a way to call an external API from update by query by iterating through each document as the calculated values might be based on each doc other field values using script??

Thanks,

2

No, there is not. I believe your best bet is to use scroll query with the calculation in your script.

3

Thanks a lot for your reply @Christian_Dahlqvist . But need help in how can I call an external function or api from within the script ? Any examples please on calling a plugin function or may be an external api by passing the current document other field values and getting a calculated value (calculation will be done externally by taking the doc values of the current document)

4

You will need to use an external script/application that runs the scroll query, determines the value snd performs the update.

5

Thank you @Christian_Dahlqvist , I am trying to use RestClient from JAVA so that get the matching documents and then update the field by externally calculating the necessaries. But now my question is can we use scroll with SearchResponse ? as I need to paginate a lots of documents?

I am reading scroll is kind of deprecated in the recent elasticsearch client 8.4. If yes what are the alternatives for better performances? as I said need to iterate through lot of documents and update each one.

Thanks,

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.