Thanks for the reply. The data we're consuming is just a SQL view that we check every few hours and only has about 10k rows in it so it's not that expensive to load imo. I don't have much control over the backend so the trigger option won't work.
In the meantime, I've also found these posts. One that can use an index template to accomplish this apparently but no details
Delete and recreate index in elasticsearch - Elastic Stack / Logstash - Discuss the Elastic Stack
And the other Delete all documents from specific ES index, that are not in the logstash feed - Elastic Stack / Logstash - Discuss the Elastic Stack which you actually suggested deleting all the documents that don't have the most recent "last updated" timestamp which makes sense. I just don't know how to approach that.
It seems these are more straight forward but just unsure if done right inside the pipeline config as well