Drop previous csv lines if field value already exists

Elastic Stack Logstash
1

image
image642×542 8.42 KB

I'd like to know if there is a way to drop/overwrite previous csv documents if a field value already exists.

In my screenshot above globalCallID is what I want to match on.

Take globalCallID_callID number 263504...

That document would only exist in Elasticsearch as:

image

2

Would it be possible to do something similar to this? Logstash - how to overwrite document instead of creating new ones

output {
    elasticsearch {
        host => 'your es host'
        action => 'update'
        document_id => "%{[globalCallID_callId]}"
        index => 'your index name
    }
}