Failure_type_logging_whitelist not working for document_missing_exception type

Elastic Stack Logstash
1

I have a pipeline set up to update documents in elasticsearch if they are found to have been removed from a database. Due to implementation details, I've decided it's easier to issue updates that fail if documents don't exist, rather than ensure documents do exist before updating them. That works fine, but it's logging WARN messages for all documents that don't exist, of course.

So, I configured the failure_type_logging_whitelist option with document_missing_exception in order to prevent that:

output {
    elasticsearch {
        index => "customers-advtest"
        hosts => ["elsearch"]
        document_id => "%{externalSystem}-%{remoteId}"
        action => update
        script => "ctx._source.status = deactivated"
        failure_type_logging_whitelist => [ "document_missing_exception" ]
    }
}

After doing so, I can see that set in the config:

[DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@failure_type_logging_whitelist = ["document_missing_exception"]

Yet I still get the WARN in the logs:

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"adv-37_1001a000197", :_index=>"customers-advtest", :_type=>"doc", :routing=>nil, :_retry_on_conflict=>1}, #<LogStash::Event:0x1a0e94dc>], :response=>{"update"=>{"_index"=>"customers-advtest", "_type"=>"doc", "_id"=>"adv-37_1001a000197", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[doc][adv-37_1001a000197]: document missing", "index_uuid"=>"qhA67Ep3R4a0NVoZLv8kHg", "shard"=>"0", "index"=>"customers-advtest"}}}}

What am I missing? Can only certain failure types be whitelisted?

2

Looking at the code, there is no test against the whitelist when it goes through the handle_dlq_status branch (i.e. a 400 or 404). One might argue that is a bug.

3

ah, I see now. the documentation should include this explanation, at least. thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.