Failure_type_logging_whitelist not working for document_missing_exception type

[jdmcalee (Justin McAleer)]

I have a pipeline set up to update documents in elasticsearch if they are found to have been removed from a database. Due to implementation details, I've decided it's easier to issue updates that fail if documents don't exist, rather than ensure documents do exist before updating them. That works fine, but it's logging WARN messages for all documents that don't exist, of course.

So, I configured the failure_type_logging_whitelist option with document_missing_exception in order to prevent that:

output {
    elasticsearch {
        index => "customers-advtest"
        hosts => ["elsearch"]
        document_id => "%{externalSystem}-%{remoteId}"
        action => update
        script => "ctx._source.status = deactivated"
        failure_type_logging_whitelist => [ "document_missing_exception" ]
    }
}

After doing so, I can see that set in the config:

[DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@failure_type_logging_whitelist = ["document_missing_exception"]

Yet I still get the WARN in the logs:

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"adv-37_1001a000197", :_index=>"customers-advtest", :_type=>"doc", :routing=>nil, :_retry_on_conflict=>1}, #<LogStash::Event:0x1a0e94dc>], :response=>{"update"=>{"_index"=>"customers-advtest", "_type"=>"doc", "_id"=>"adv-37_1001a000197", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[doc][adv-37_1001a000197]: document missing", "index_uuid"=>"qhA67Ep3R4a0NVoZLv8kHg", "shard"=>"0", "index"=>"customers-advtest"}}}}

What am I missing? Can only certain failure types be whitelisted?

[Badger]

Looking at the code, there is no test against the whitelist when it goes through the handle_dlq_status branch (i.e. a 400 or 404). One might argue that is a bug.

[jdmcalee (Justin McAleer)]

ah, I see now. the documentation should include this explanation, at least. thanks