# Reconcile data in ElasticSearch using logstash filter or output plugin

**URL:** <https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317>\
**Category:** Logstash\
**Created:** [July 18, 2024, 6:00am UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317 "2024-07-18T06:00:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vvavad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvavad/32/136129_2.png) [@vvavad](https://discuss.elastic.co/u/vvavad)\
**Post date:** [July 18, 2024, 6:00am UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317/1 "2024-07-18T06:00:47Z")

</div>

I have a Elasticsearch index(index1) with document structure

```auto
{
"id": "id1", 
"addresses": [
{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"},
], 
"field1": "f1", "field2": "f2", "field3": "f3" 
}

```

I have a logstash job/pipeline that will ingest data from json file and push it to elasticSearch index(index2). The data in json file:

```auto
{
"id": "a1", 
"addresses": [
{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"},
{"address": "a21","address_2": "a22","city": "c2","state": "s2","zip": "12346","phone": "9191919192"},
], 
"field5": [{"f51", "f52",}], "field6": "f6" 
}

```

I wanted to check if logstash pipeline can be used to read the json file and update data in the existing elasticSearch index(index1), instead of writing the data to a new index(index2).  
Another question is, is there a way to reconcile data in the logstash job/pipeline. Reconciliation could be of the existing addresses in the Elasticsearch index(index1) and addresses coming from json file thats getting read based of the unique field "id" which is also id of the document.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [August 8, 2024, 5:30pm UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317/2 "2024-08-08T17:30:14Z")

</div>

Setting the `action` on the Elasticsearch output to `index` will update the document (by id) in Elasticsearch if it finds a document already exists and will create a new document if it finds that it does not already exist.

---

<div class="post-metadata">

**Author:** ![vvavad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvavad/32/136129_2.png) [@vvavad](https://discuss.elastic.co/u/vvavad)\
**Post date:** [August 9, 2024, 12:27pm UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317/3 "2024-08-09T12:27:21Z")

</div>

Thanks for the response.  
The challenge is, the existing elasticSearch index1 and the data in incoming json file is different. I only want to add/update a few fields in the index.  
Whats happening is its replacing the document for a given id with data from json, the original data from index1 for that id is lost.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [August 9, 2024, 1:46pm UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317/4 "2024-08-09T13:46:11Z")

</div>

You could use the Elasticsearch Filter plugin to query for the existing document and add fields to the document before sending it to the output: [Elasticsearch filter plugin | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)

i.e.

1. Read JSON object
2. Query document ID from Elasticsearch
3. Add couple of fields from JSON object to document queried from Elastisearch
4. Output updated document to Elasticsearch

---

<div class="post-metadata">

**Author:** ![vvavad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvavad/32/136129_2.png) [@vvavad](https://discuss.elastic.co/u/vvavad)\
**Post date:** [August 10, 2024, 11:13am UTC](https://discuss.elastic.co/t/reconcile-data-in-elasticsearch-using-logstash-filter-or-output-plugin/363317/5 "2024-08-10T11:13:27Z")

</div>

Thanks. This is what I am also trying. Will keep this thread updated once I achieve this.