Logstash upsert usage

Elastic Stack Logstash
1

I am merging two docs based on a ID which i frame it out from the document i receive. The two different doc which i want merge share the same ID and i merge it in logstash using upsert. The document are merging but at the same time error are found in logstash logs. One example is below

[2019-02-13T09:30:06,364][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>409, :action=>["update", {:_id=>"beta-32104172.21.249.521.2.840.113837.73102298.1540581278.61.2.392.200036.9116.2.6.1.3268.2046999618.1540534879.921419iSyntaxInitImageTQ=0&sid=b4be6021-d8af-41c2-a114-a35a7ee7cb78", :_index=>"beta-2019-02", :_type=>"doc", :_routing=>nil, :_retry_on_conflict=>1}, #LogStash::Event:0x2634f921], :response=>{"update"=>{"_index"=>"beta-2019-02", "_type"=>"doc", "_id"=>"beta-32104172.21.249.521.2.840.113837.73102298.1540581278.61.2.392.200036.9116.2.6.1.3268.2046999618.1540534879.921419iSyntaxInitImageTQ=0&sid=b4be6021-d8af-41c2-a114-a35a7ee7cb78", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[doc][beta-32104172.21.249.521.2.840.113837.73102298.1540581278.61.2.392.200036.9116.2.6.1.3268.2046999618.1540534879.921419iSyntaxInitImageTQ=0&sid=b4be6021-d8af-41c2-a114-a35a7ee7cb78]: version conflict, current version [3] is different than the one provided [2]", "index_uuid"=>"yvM1Rv1qS0qXs-SB4YHnyw", "shard"=>"0", "index"=>"beta-2019-02"}}}}

my logstash output plugin is as below

if [uri_query] and ![Server_ElapsedMS] {
elasticsearch {
hosts => ["abcd:9200"]
index => 'beta-%{+YYYY-MM}'
action => "update"
doc_as_upsert => true
document_id => "beta-%{ThreadID}%{SAHost}%{StudyUID}%{ImageInstanceUID}%{StudyAuthorityCallType}%{uri_query}"
}
}
if [uri_query] and [Server_ElapsedMS] {
elasticsearch {
hosts => ["abcd:9200"]
index => 'beta-%{+YYYY-MM}'
action => "update"
doc_as_upsert => true
document_id => "beta-%{ThreadID}%{SAHost}%{StudyUID}%{ImageInstanceUID}%{StudyAuthorityCallType}%{uri_query}"
}
}
Can any one help out in resolving this.

2

That message is not an error; it is a warning. If the behaviour is working as you expect (e.g., when the upsert is retried, a merge occurs), then you can ignore the warning.

1 Like
3

Thanks for the update. Is there any way i can prevent this message being logged. It fills up my log files almost...

4

You might be looking for the failure_type_logging_whitelist => [409] directive :slight_smile:

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.