I have documents with known unique IDs (document_id) in Elasticsearch. I am using Filebeat to ship to Logstash and then to Elasticsearch. I would like to add fields from documents of a certain type coming from Filebeat to existing documents in Elasticsearch if ref_id field in incoming document is equal to the document_id of an existing document. How can this be done?
For example I have a document that has a state associated with it. A separate file contains state changes and a state date. Ultimately, I would only make the change to the existing documents fields if the state date is newer than the state date of the state already in the document.