# How to pick up certain Logstash events so I can ignore/work on them

**URL:** <https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935>\
**Category:** Logstash\
**Created:** [February 7, 2023, 8:47pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935 "2023-02-07T20:47:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![SamuelSMendes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samuelsmendes/32/104246_2.png) [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\
**Post date:** [February 7, 2023, 8:47pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/1 "2023-02-07T20:47:29Z")

</div>

So I've been working with a Logstash pipeline which deals with creating and updating a few documents. And when the schedule hits and the creation repeats the following line pop up:

```auto
[2023-02-07T17:36:07,915][WARN][logstash.outputs.elasticsearch][main][5fc5d1856723xxxxxxxxxce5db3e37e8390c07d899f2e5a85b95ae09a7e85ed6] Failed action {:status=>409, etc, etc,"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception"}

```

I can understand the reason this keeps happening but I'd like to have a way to treat it. Maybe to just ignore these with a `drop` or something similar to it. I've checked a few ways to grab Logstash events with the [Ruby](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) filter plugin. But nothing that reached this line of status/error which I'm interested at.

Is there a proper way to approach this?

Best regards.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 8, 2023, 12:54am UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/2 "2023-02-08T00:54:01Z")

</div>

This error is already at the output block of the pipeline and you can't catch it.

But 409 errors logs a warn and are dropped, there is no retry when Elasticsearch respond with an 409 error.

You probably have some kind of race condition in your pipeline trying to create/update the same document id, this [post](https://discuss.elastic.co/t/version-conflict-409-question/311335/4) may explain better what could cause an 409 error.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 8, 2023, 1:57am UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/3 "2023-02-08T01:57:11Z")

</div>

I do not think a single logstash instance can produce this exception by itself. If elasticsearch reads a document to do an update, it double checks the version when it writes the document back to the index. If the version in the index is not the version it updated then someone else sneaked in an update ahead of it, and it generates this exception.

I believe a logstash output is single threaded, I see no way for it to cause two parallel updates to the same document.

It could be a second logstash instance, or something else calling the API.

---

<div class="post-metadata">

**Author:** ![SamuelSMendes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samuelsmendes/32/104246_2.png) [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\
**Post date:** [February 8, 2023, 12:15pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/4 "2023-02-08T12:15:50Z")

</div>

@leandrojmp and @Badger, thanks for the answer. The thing here is not exactly the "why" it is happening. It is exactly as you two have said. The documents are being created and updated at the same pace.

Here is the output that is making it happen:

```auto
output {

    elasticsearch{
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "create"
    }

    elasticsearch{
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "update"
    }

    #stdout { codec => rubydebug }
}

```

The reasoning behind this config is that, if a document with certain id doesn't exists it is created, and if it does it can't be created and is updated instead.

By the reaction here I now feel that it may not have been the best way to get this result...(which works despite the error logs I showed before).

If there is another way to approach this I'll be glad to know. At the moment I still coulnd't find another way to get this done.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 8, 2023, 12:25pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/5 "2023-02-08T12:25:00Z")

</div>

> [@SamuelSMendes](#):
>
> If there is another way to approach this I'll be glad to know. At the moment I still coulnd't find another way to get this done.

This is not the correct approach, check the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action) for the `action` option.

You have this:

> `update` : updates a document by id. **Update has a special case where you can upsert — update a document if not already present**. See the `doc_as_upsert` option

And for the [doc\_as\_upsert](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-doc_as_upsert) documentation you have this:

> Enable `doc_as_upsert` for update mode. Create a new document with source if `document_id` doesn’t exist in Elasticsearch.

You need to remove the output with the `create` action and add this in the output with the `update` action.

```auto
    elasticsearch {
        index => "xxxx"
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "xxxxxxx"
        document_id => "%{[ticket][key]}"
        action => "update"
        doc_as_upsert => true
    }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2023, 12:25pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935/6 "2023-03-08T12:25:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.