I have an index in elastic, which is fed by two pipelines.
pipeline 1 provides the base data for the document
fields: id, name, number
pipeline 2 needs to add new fields to the existing document based on the ID.
fields: id, color, comment
When pipeline 1 sends data, the document looks like this:
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567' }
When pipeline 2 sends data, it overwrites the original document (based on fingerprint of id), and removes the existing data.
So now the document looks like:
{ 'id': '1234', 'color': 'red', 'comment': 'he likes red' }
I understand that elastic doesn't actually update, instead it deletes the existing document, and replaces it.
But is there some way I can tell logstash, to tell elastic that "take the data already stored, and combine it with this new data, and create a new document".
So my final document should look like this:
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567', 'color': 'red', 'comment': 'he likes red' }
thanks in advance for any help.