# Sending new fields to exisiting Elastic document, losing old data

**URL:** <https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992>\
**Category:** Logstash\
**Created:** [November 7, 2019, 4:46pm UTC](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992 "2019-11-07T16:46:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![EdgeSync](https://avatars.discourse-cdn.com/v4/letter/e/90ced4/32.png) [@EdgeSync](https://discuss.elastic.co/u/EdgeSync)\
**Post date:** [November 7, 2019, 4:46pm UTC](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992/1 "2019-11-07T16:46:42Z")

</div>

I have an index in elastic, which is fed by two pipelines.

pipeline 1 provides the base data for the document  
fields: id, name, number

pipeline 2 needs to add new fields to the existing document based on the ID.  
fields: id, color, comment

When pipeline 1 sends data, the document looks like this:  
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567' }

When pipeline 2 sends data, it overwrites the original document (based on fingerprint of id), and removes the existing data.  
So now the document looks like:  
{ 'id': '1234', 'color': 'red', 'comment': 'he likes red' }

I understand that elastic doesn't actually update, instead it deletes the existing document, and replaces it.

But is there some way I can tell logstash, to tell elastic that "take the data already stored, and combine it with this new data, and create a new document".  
So my final document should look like this:  
{ 'id': '1234', 'name': 'john smith', 'number': '123-345-567', 'color': 'red', 'comment': 'he likes red' }

thanks in advance for any help.

---

<div class="post-metadata">

**Author:** ![EdgeSync](https://avatars.discourse-cdn.com/v4/letter/e/90ced4/32.png) [@EdgeSync](https://discuss.elastic.co/u/EdgeSync)\
**Post date:** [November 7, 2019, 5:30pm UTC](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992/2 "2019-11-07T17:30:07Z")

</div>

I have solved it myself 😃

based on this post: [doc\_as\_upsert vs upsert properties](https://discuss.elastic.co/t/update-ops-doc-as-upsert-vs-upsert-properties/14108)

It was required to make it an upsert.

so my logstash config looks something like this

> input {  
> udp {  
> port =\> 10001  
> codec =\> json  
> }  
> }
> 
> filter {  
> fingerprint {  
> source =\> "id"  
> target =\> "[@metadata][fingerprint]"  
> method =\> "MURMUR3"  
> }  
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["[https://127.0.0.1:9200](https://127.0.0.1:9200)"]  
> user =\> "logstash\_account"  
> password =\> "some\_password\_here"  
> cacert =\> "/etc/logstash/conf.d/cacert.cer"  
> ssl\_certificate\_verification =\> false  
> document\_id =\> "%{[@metadata][fingerprint]}"  
> index =\> "some\_index"  
> doc\_as\_upsert =\> true  
> action =\> update  
> }  
> }

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2019, 5:30pm UTC](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992/3 "2019-12-05T17:30:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.