How to pick up certain Logstash events so I can ignore/work on them

[SamuelSMendes (Samuel S. Mendes Soares)]

So I've been working with a Logstash pipeline which deals with creating and updating a few documents. And when the schedule hits and the creation repeats the following line pop up:

[2023-02-07T17:36:07,915][WARN ][logstash.outputs.elasticsearch][main][5fc5d1856723xxxxxxxxxce5db3e37e8390c07d899f2e5a85b95ae09a7e85ed6] Failed action {:status=>409, etc, etc,"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception"}

I can understand the reason this keeps happening but I'd like to have a way to treat it. Maybe to just ignore these with a drop or something similar to it. I've checked a few ways to grab Logstash events with the Ruby filter plugin. But nothing that reached this line of status/error which I'm interested at.

Is there a proper way to approach this?

Best regards.

[leandrojmp (Leandro Pereira)]

This error is already at the output block of the pipeline and you can't catch it.

But 409 errors logs a warn and are dropped, there is no retry when Elasticsearch respond with an 409 error.

You probably have some kind of race condition in your pipeline trying to create/update the same document id, this post may explain better what could cause an 409 error.

[Badger]

I do not think a single logstash instance can produce this exception by itself. If elasticsearch reads a document to do an update, it double checks the version when it writes the document back to the index. If the version in the index is not the version it updated then someone else sneaked in an update ahead of it, and it generates this exception.

I believe a logstash output is single threaded, I see no way for it to cause two parallel updates to the same document.

It could be a second logstash instance, or something else calling the API.

[SamuelSMendes (Samuel S. Mendes Soares)]

@leandrojmp and @Badger, thanks for the answer. The thing here is not exactly the "why" it is happening. It is exactly as you two have said. The documents are being created and updated at the same pace.

Here is the output that is making it happen:

output {

    elasticsearch{
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "create"
    }

    elasticsearch{
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "update"
    }

    #stdout { codec => rubydebug }
}

The reasoning behind this config is that, if a document with certain id doesn't exists it is created, and if it does it can't be created and is updated instead.

By the reaction here I now feel that it may not have been the best way to get this result...(which works despite the error logs I showed before).

If there is another way to approach this I'll be glad to know. At the moment I still coulnd't find another way to get this done.

[leandrojmp (Leandro Pereira)]

@SamuelSMendes

If there is another way to approach this I'll be glad to know. At the moment I still coulnd't find another way to get this done.

This is not the correct approach, check the documentation for the action option.

You have this:

update : updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the doc_as_upsert option

And for the doc_as_upsert documentation you have this:

Enable doc_as_upsert for update mode. Create a new document with source if document_id doesn’t exist in Elasticsearch.

You need to remove the output with the create action and add this in the output with the update action.

    elasticsearch {
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "update"
        doc_as_upsert => true
    }