Unsure if I'm doing something wrong, but I have the following ES output setup:
output {
stdout { codec => json_lines }
elasticsearch {
hosts => "search:9200"
index => "posts"
template => "/indexer/templates/posts.json"
action => "%{[@metadata][index_action]}"
document_id => "%{id}"
retry_on_conflict => 5
}
}
However I’m still running into conflicts, regardless of how high I set the count to:
indexer_2 | [2018-02-08T03:46:37,376][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>409, :action=>["update", {:_id=>"107305507", :_index=>"posts", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x69a55269>], :response=>{"update"=>{"_index"=>"posts", "_type"=>"doc", "_id"=>"107305507", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[doc][107305507]: version conflict, current version [16] is different than the one provided [15]", "index_uuid"=>"UUKBo3L1QLWj-wua6eJs6g", "shard"=>"0", "index"=>"posts"}}}}
I setup a proxy in between Logstash/ES to see what the plugin was posting, and I'm not seeing the retry value anywhere:
search_proxy_1 | 200 POST /_bulk \n{"update":{"_id":"107305507","_index":"posts","_type":"doc","_routing":null}}\n{"doc":{"@version":"1","id":107305507,"metrics":{"likes":624122,"comments":5445},"@timestamp":"2018-02-08T03:46:38.495Z"}}\n{"update":{"_id":"107305512","_index":"posts","_type":"doc","_routing":null}}\n{"doc":{"@version":"1","id":107305512,"metrics":{"likes":434569,"comments":5257},"@timestamp":"2018-02-08T03:46:38.526Z"}}\n
Unless I'm mistaken, I'd expect to see "_retry_on_conflict": 5 in the update JSON.
I'm running logstash 6.2.0, elasticsearch 6.0.1 and logstash-output-elasticsearch 9.0.2.
Appreciate any insight any of you can lend here!