Populate new field in elastic index using key of the index

[bipin_kumar2 (bipin kumar)]

Hi ,
I have index which of very large size and in populate incremental data into this index by using upsert into hive external table using id of index pointing to elastic index , now i have to add a new field into this index and have to populate that field for historical records using the id , i dont want to refresh complete index again as it takes lots of time to do that so i want to create a table in hive with id and new field and using that table i want to populate only newley added field in index , or i can also that that table into a new index in elastic and want use that new index to populate historical data for this new field . Do we have any approach to achieve this . Thanks

[warkolm (Mark Walkom)]

Welcome to our community!

You can add the new fields to the new documents, and the old ones will just have null values for them.
If you want the old records to have these values you need to update the documents.

[bipin_kumar2 (bipin kumar)]

if i have a index with id and new value id values are same for both indexes , how can we update all document at once with new value using key , basically i am asking to to achieve update by join like sql in elastic as elastic have no features like join and update , i want to move only id column and new column values from hive to elastic i don't want to repopulate complete index as it has 300 columns and it takes lot of time to move full data from hive to elastic.

[Christian_Dahlqvist]

If you want to add fields to the documents the full documents will need to be reindexed. As far as I know there is no shortcut.