Insert to elasticsearch from logstash IF NOT EXISTS

Elastic Stack Logstash
1

Hi,

I currently have an index that I set unique ids and sometimes there is replica data. In my use case, older data is often more accurate than newer data. So I would like to have logstash only insert if the _id currently does not exist in the index. If it exists, the event shouldn't be sent to Elasticsearch and if it doesn't exist it should be sent. Any thoughts on how I can do this?

2

Set the action option on the Elasticsearch output to "create" -- create : indexes a document, fails if a document by that id already exists in the index. Hopefully logstash does not endlessly retry :slight_smile:

3

Great, I'll give that a shot!

4

looks like it worked, but yes logtash retries tirelessly. No problem because my data isn't streaming, just one time uploads

thanks.

5

If it does indeed retry an event that is rejected because it already exists then that feels very much like a bug. @warkolm, do you agree?

6

Sounds like it, yeah.

7

What happens specifically is that when I'm running it as a command (not as a service), it will run through it, then terminate the pipeline and start the pipeline all over again continually.

8

It does not retry the 409, there is just a single instance of

[2022-06-08T14:08:45,101][WARN ][logstash.outputs.Elasticsearch][main][ded7c2896208304e6887fb9969cdd51534a5ace0eafa08289f565caac9250dbd] Failed action {:status=>409, :action=>["create", {:_id=>"14", :_index=>"ecs-logstash-2022.06.08", :routing=>nil}, {"event"=>{"sequence"=>0, "original"=>"{"id": 14}"}, "@version"=>"1", "id"=>14, "@timestamp"=>2022-06-08T18:08:44.971913Z, "host"=>{"name"=>"ip-172-31-22-149.us-east-2.compute.internal"}}], :response=>{"create"=>{"_index"=>"ecs-logstash-2022.06.08", "_id"=>"14", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[14]: version conflict, document already exists (current version [1])", "index_uuid"=>"MiVA-Ee_QfOJhn_C3ckXlA", "shard"=>"0", "index"=>"ecs-logstash-2022.06.08"}}}}

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.