Logstash update if id matches

Elastic Stack Logstash
1

is there a possibility to update a document if id matches else insert ?
Alright i'm reading a kafka topic and this a sample of my topic

`

> "schema":{"type":"struct","fields":[{"type":"string","optional":true,"field":"name"},{"type":"string","optional":true,"field":"season"},{"type":"string","optional":true,"field":"id"}],"optional":false,"name":"mongo_conn.test.testCollection"},"payload":{"name":"aothertest","season":"reallyweird","id":"5d0a2a75baff7c1e89ef44d3"}}

`

ps: all i need is found under payload [name,season,id]

and this is my logstash input

input {
  kafka {
        bootstrap_servers => "localhost:9092"
        topics => "mongo_conn.test.testCollection"
        auto_offset_reset => "earliest"
codec => json{}
}
}

for example incase if payload.id matches a already existing payload.id , update the document

2

If you are using an elasticsearch output then you can use doc_as_upsert.

3

@Badger I've added that, but nothing happens it keeps on pushing the update as a new insert maybe i need to specify the id nested in payload ? as i mentioned before all my info are inside payload the rest i don't need it's just the way the format is in kafka

4

You are using

document_id => "%{[payload][id]}"

right?

1 Like
5

@Badger much love, no i was not using that i just tested it and it's working fine, i have a question thought what about the action =update thing ?

6

I believe if you are setting doc_as_upsert then you should be using action => "update".

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.