Reconcile data in ElasticSearch using logstash filter or output plugin

Elastic Stack Logstash
1

I have a Elasticsearch index(index1) with document structure

{
"id": "id1", 
"addresses": [
{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"},
], 
"field1": "f1",  "field2": "f2", "field3": "f3" 
}

I have a logstash job/pipeline that will ingest data from json file and push it to elasticSearch index(index2). The data in json file:

{
"id": "a1", 
"addresses": [
{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"},
{"address": "a21","address_2": "a22","city": "c2","state": "s2","zip": "12346","phone": "9191919192"},
], 
"field5": [{"f51", "f52",}], "field6": "f6" 
}

I wanted to check if logstash pipeline can be used to read the json file and update data in the existing elasticSearch index(index1), instead of writing the data to a new index(index2).
Another question is, is there a way to reconcile data in the logstash job/pipeline. Reconciliation could be of the existing addresses in the Elasticsearch index(index1) and addresses coming from json file thats getting read based of the unique field "id" which is also id of the document.

2

Setting the action on the Elasticsearch output to index will update the document (by id) in Elasticsearch if it finds a document already exists and will create a new document if it finds that it does not already exist.

1 Like
3

Thanks for the response.
The challenge is, the existing elasticSearch index1 and the data in incoming json file is different. I only want to add/update a few fields in the index.
Whats happening is its replacing the document for a given id with data from json, the original data from index1 for that id is lost.

4

You could use the Elasticsearch Filter plugin to query for the existing document and add fields to the document before sending it to the output: Elasticsearch filter plugin | Logstash Reference [8.15] | Elastic

i.e.

  1. Read JSON object
  2. Query document ID from Elasticsearch
  3. Add couple of fields from JSON object to document queried from Elastisearch
  4. Output updated document to Elasticsearch
1 Like
5

Thanks. This is what I am also trying. Will keep this thread updated once I achieve this.