Logstash aggregating through different events

Elastic Stack Logstash
1

Hello,

I have a question regarding aggregating with logstash
my usecase is that I a csv file with specific attrbuites but some of the attrbuites come empty
then I receive another file that contain these missing attrbuite
so my question is how can I approach this + if I don't have a unique field what can I do

2

Missing or empty field you can fill with null or "" or delete, it's up to you. I would make decision to unify for all field base on file content, what is important for the visualization or search. With few IFs you can handle several cases:

if ![fieldA] { #if no exist, add field without value
mutate { add_field => { "fieldA" => "" } }
}
if ![fieldA] { #if no exist, add field with null 
ruby { code => "event['fieldA'] = nil" }
}
if [fieldA]=="" { #if is empty, set fieldA to null 
ruby { code => "event['fieldA'] = nil" }
}

If unique value doesn't exist, use fingerprint plugin with method MD5 or UUID on the message. You can use 1 or more fields to make an unique field with value for document_id => "%{fingerprintuuid}".

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.