I have two indices, one contains event records that contain a timestamp, location, user id and country, the other contains a summary of all the event records. The summary includes last timestamp, last location and last country. It also has an array of countries visited.
I've set up a transform that takes the latest event (data can arrive out of order) and writes it to the second index. Since the user id is the unique field we key off of, the same document gets repeatedly updated as new events come in. Currently, we have an external app that parses all the data and does upserts against the summary index and includes a script to add new countries to the countries visited array.
We are trying to migrate this to an ingest pipeline, but my understanding is that the ingest context can only see the record being written and not the existing one. Is there any way to reference the existing document? Or some other way to do this, we are trying to sunset the legacy app and move this functionality in to Elasticsearch.