# Exclude version conflict, document already exists from logstash.log

**URL:** <https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186>\
**Category:** Logstash\
**Created:** [January 31, 2024, 1:05pm UTC](https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186 "2024-01-31T13:05:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [January 31, 2024, 1:05pm UTC](https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186/1 "2024-01-31T13:05:28Z")

</div>

Hey Everyone,

I'm having some issues with too much noise in my Logstash logs.  
What's happening is because I'm using the [Threat Intel integrations](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html) it's spamming my logs with `version conflict, document already exists` WARN logs.

The reason for that is it queries specific APIs for new data, and makes sure there are no duplicates by generating a doc\_id. Since almost all of the IoCs will already be present after the first run, it's absolutely filling my Logstash logs with the errors like below:

```auto
{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1706705848184,"thread":"[es-agent-output]>worker2","logEvent":{"message":"Failed action","status":409,"action":["create",{"_index":"logs-ti_otx.threat-prod"},{"data_stream":{"type":"logs","dataset":"ti_otx.threat","namespace":"prod"},"elastic_agent":{"version":"8.11.3","snapshot":false,"id":"a19710f3-f2f9-48d3-b75d-69dfee8214f5"},"tags":["forwarded","otx-threat"],"ecs":{"version":"8.0.0"},"@timestamp":"2024-01-31T12:50:34.747Z","@version":"1","input":{"type":"httpjson"},"event":{"created":"2024-01-31T12:50:34.747Z","dataset":"ti_otx.threat"},"type":"elastic-agent","agent":{"ephemeral_id":"6a773169-d743-451e-842f-cb1d001cba3d","version":"8.11.3","name":"cs-srv-elk03","type":"filebeat","id":"a19710f3-f2f9-48d3-b75d-69dfee8214f5"},"message":"{\"count\":47,\"next\":null,\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":3831864571,\"indicator\":\"ca49787e7ea3b81fccca2ae45852a3d6\",\"title\":null,\"type\":\"FileHash-MD5\"}}"}],"response":{"create":{"status":409,"error":{"type":"version_conflict_engine_exception","reason":"[ZgjDoPgwQvMm/jZ3uuZthO0cQHk=]: version conflict, document already exists (current version [1])","index_uuid":"UK1EJqgmTq6DycwUZpJvYw","shard":"0","index":".ds-logs-ti_otx.threat-prod-2024.01.31-000008"}}}}}

```

Is there any way I can stop a specific log from showing up? I know I can just bump up the verbosity to `ERROR` but that's not an acceptable solution.

Here's my ES output if anything can be tweaked here.

```auto
output {
  elasticsearch {
    hosts => ["..."]
    data_stream => true
    data_stream_auto_routing => "true"
    ssl_enabled => true
    cacert => "/etc/logstash/certs/ca.crt"
    user => "logstash_writer"
    password => "..."
    manage_template => false
    action => "create"
  }
}

```

Thanks for any help in advance!

Cheers,  
Luka

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 31, 2024, 1:53pm UTC](https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186/2 "2024-01-31T13:53:24Z")

</div>

> [@lduvnjak](#):
>
> Is there any way I can stop a specific log from showing up?

You are using data streams, you cannot update a data document from Logstash if you are using data streams, you would need to change to using normal indices and then you would be able to upsert a document so if the document exists, it will be updated.

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [January 31, 2024, 3:42pm UTC](https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186/3 "2024-01-31T15:42:08Z")

</div>

Does that mean I'd have to have separate output configurations depending on if the dataset is from a TI integration?

Would I also need to create a custom Index Template with the Data Stream option set to `false`? Considering the default one has it enabled:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/3/6366fdc3aba39bbce49cd3174f9393d2258b60dc.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2024, 3:42pm UTC](https://discuss.elastic.co/t/exclude-version-conflict-document-already-exists-from-logstash-log/352186/4 "2024-02-28T15:42:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.