Sending new fields to exisiting Elastic document, losing old data

Elastic Stack Logstash
1

I have an index in elastic, which is fed by two pipelines.

pipeline 1 provides the base data for the document
fields: id, name, number

pipeline 2 needs to add new fields to the existing document based on the ID.
fields: id, color, comment

When pipeline 1 sends data, the document looks like this:
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567' }

When pipeline 2 sends data, it overwrites the original document (based on fingerprint of id), and removes the existing data.
So now the document looks like:
{ 'id': '1234', 'color': 'red', 'comment': 'he likes red' }

I understand that elastic doesn't actually update, instead it deletes the existing document, and replaces it.

But is there some way I can tell logstash, to tell elastic that "take the data already stored, and combine it with this new data, and create a new document".
So my final document should look like this:
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567', 'color': 'red', 'comment': 'he likes red' }

thanks in advance for any help.

2

I have solved it myself :smiley:

based on this post: doc_as_upsert vs upsert properties

It was required to make it an upsert.

so my logstash config looks something like this

input {
udp {
port => 10001
codec => json
}
}

filter {
fingerprint {
source => "id"
target => "[@metadata][fingerprint]"
method => "MURMUR3"
}
}

output {
elasticsearch {
hosts => ["https://127.0.0.1:9200"]
user => "logstash_account"
password => "some_password_here"
cacert => "/etc/logstash/conf.d/cacert.cer"
ssl_certificate_verification => false
document_id => "%{[@metadata][fingerprint]}"
index => "some_index"
doc_as_upsert => true
action => update
}
}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.