How can I index only new documents without updating the older ones?

[SamuelSMendes (Samuel S. Mendes Soares)]

I am aware of the create action but when I use it a horrendous WARN log is printed in the logstash screen. The solution of create would fit perfect if it wasn't for it. So I've been wondering if there is another way to approach this case.

This is the log I want to get rid off by the way:

[2023-06-20T12:31:33,383][WARN ][logstash.outputs.elasticsearch][devices_apps][a8f9564896360f471059c7c3b107c8cf03e5a0274a1f4e663ea2a1ce836b72bb] Failed action {:status=>409, :action=>["create", {:_id=>"b1a63b24/com.xxxx.calculator", :_index=>"devices", :routing=>nil}, {"access_token"=>"gjysrkjskdhkukedk", "@version"=>"1", "request"=>"completo", "body"=>{"version"=>"12.3.81", "name"=>"Calculator", "identifier"=>"com.miui.calculator", "tags"=>["system"], "buildNumber"=>"201203681", "icon"=>nil}, "tags"=>["app"], "@timestamp"=>2023-06-20T15:30:06.489467654Z, "device"=>{"id"=>"b1a63b24"}}], :response=>{"create"=>{"_index"=>"devices", "_id"=>"b1a63b24/com.miui.calculator", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[b1a63b24/com.miui.calculator]: version conflict, document already exists (current version [1])", "index_uuid"=>"dQw4wZ2eQZScKG300QKMCw", "shard"=>"0", "index"=>"devices"}}}}

Appreciate the help.

[leandrojmp (Leandro Pereira)]

If you want to insert only new documents without updating older documents you need to use create.

To not see this error you would have two options basically, one would be to query your elasticsearch if the custom id you are using already exists and drop the document if it exists, but this can be pretty expensive.

If the noise warn logs is your issue the other option would be to change the log level of the elasticsearch output to ERROR to suppress those WARN logs, but this will suppress ALL WARN logs for ALL elasticsearch outputs.

You would just need to run this command in your Logstash server:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'{ "logger.logstash.outputs.elasticsearch" : "ERROR" }'