Exclude version conflict, document already exists from logstash.log

Elastic Stack Logstash
1

Hey Everyone,

I'm having some issues with too much noise in my Logstash logs.
What's happening is because I'm using the Threat Intel integrations it's spamming my logs with version conflict, document already exists WARN logs.

The reason for that is it queries specific APIs for new data, and makes sure there are no duplicates by generating a doc_id. Since almost all of the IoCs will already be present after the first run, it's absolutely filling my Logstash logs with the errors like below:

{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1706705848184,"thread":"[es-agent-output]>worker2","logEvent":{"message":"Failed action","status":409,"action":["create",{"_index":"logs-ti_otx.threat-prod"},{"data_stream":{"type":"logs","dataset":"ti_otx.threat","namespace":"prod"},"elastic_agent":{"version":"8.11.3","snapshot":false,"id":"a19710f3-f2f9-48d3-b75d-69dfee8214f5"},"tags":["forwarded","otx-threat"],"ecs":{"version":"8.0.0"},"@timestamp":"2024-01-31T12:50:34.747Z","@version":"1","input":{"type":"httpjson"},"event":{"created":"2024-01-31T12:50:34.747Z","dataset":"ti_otx.threat"},"type":"elastic-agent","agent":{"ephemeral_id":"6a773169-d743-451e-842f-cb1d001cba3d","version":"8.11.3","name":"cs-srv-elk03","type":"filebeat","id":"a19710f3-f2f9-48d3-b75d-69dfee8214f5"},"message":"{\"count\":47,\"next\":null,\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":3831864571,\"indicator\":\"ca49787e7ea3b81fccca2ae45852a3d6\",\"title\":null,\"type\":\"FileHash-MD5\"}}"}],"response":{"create":{"status":409,"error":{"type":"version_conflict_engine_exception","reason":"[ZgjDoPgwQvMm/jZ3uuZthO0cQHk=]: version conflict, document already exists (current version [1])","index_uuid":"UK1EJqgmTq6DycwUZpJvYw","shard":"0","index":".ds-logs-ti_otx.threat-prod-2024.01.31-000008"}}}}}

Is there any way I can stop a specific log from showing up? I know I can just bump up the verbosity to ERROR but that's not an acceptable solution.

Here's my ES output if anything can be tweaked here.

output {
  elasticsearch {
    hosts => ["..."]
    data_stream => true
    data_stream_auto_routing => "true"
    ssl_enabled => true
    cacert => "/etc/logstash/certs/ca.crt"
    user => "logstash_writer"
    password => "..."
    manage_template => false
    action => "create"
  }
}

Thanks for any help in advance!

Cheers,
Luka

2

You are using data streams, you cannot update a data document from Logstash if you are using data streams, you would need to change to using normal indices and then you would be able to upsert a document so if the document exists, it will be updated.

3

Does that mean I'd have to have separate output configurations depending on if the dataset is from a TI integration?

Would I also need to create a custom Index Template with the Data Stream option set to false? Considering the default one has it enabled:

image
image604×548 31.6 KB

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.