Elasticsearch output not respecting retry_on_conflict


(Shaun Harrison) #1

Unsure if I'm doing something wrong, but I have the following ES output setup:

output {
	stdout { codec => json_lines }
	elasticsearch {
		hosts => "search:9200"
		index => "posts"
		template => "/indexer/templates/posts.json"
		action => "%{[@metadata][index_action]}"
		document_id => "%{id}"
		retry_on_conflict => 5
	}
}

However I’m still running into conflicts, regardless of how high I set the count to:

indexer_2       | [2018-02-08T03:46:37,376][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>409, :action=>["update", {:_id=>"107305507", :_index=>"posts", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x69a55269>], :response=>{"update"=>{"_index"=>"posts", "_type"=>"doc", "_id"=>"107305507", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[doc][107305507]: version conflict, current version [16] is different than the one provided [15]", "index_uuid"=>"UUKBo3L1QLWj-wua6eJs6g", "shard"=>"0", "index"=>"posts"}}}}

I setup a proxy in between Logstash/ES to see what the plugin was posting, and I'm not seeing the retry value anywhere:

search_proxy_1  | 200 POST /_bulk \n{"update":{"_id":"107305507","_index":"posts","_type":"doc","_routing":null}}\n{"doc":{"@version":"1","id":107305507,"metrics":{"likes":624122,"comments":5445},"@timestamp":"2018-02-08T03:46:38.495Z"}}\n{"update":{"_id":"107305512","_index":"posts","_type":"doc","_routing":null}}\n{"doc":{"@version":"1","id":107305512,"metrics":{"likes":434569,"comments":5257},"@timestamp":"2018-02-08T03:46:38.526Z"}}\n

Unless I'm mistaken, I'd expect to see "_retry_on_conflict": 5 in the update JSON.

I'm running logstash 6.2.0, elasticsearch 6.0.1 and logstash-output-elasticsearch 9.0.2.

Appreciate any insight any of you can lend here!


(Shaun Harrison) #2

This actually looks to be a bug, I've opened an issue here: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/731


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.