Logstash/grok to match only first occurrence and stop parsing repeatedly for same values

Elastic Stack Logstash
3

If that is really what you want (one document for each value of temperature) then use the temperature as the document_id (assuming you are sending data to elasticsearch). If you are not using elasticsearch you could do it in a ruby filter using something like

ruby {
    init => '@seenValues = {}'
    code => '
        value = event.get("someField")
        if @seenValues.include? (value)
            event.cancel
        end
        @seenValues[value] = 1
    '
}

If you only want events where that field changes then it can also be done using ruby. Something like

ruby {
    init => '@lastValue = ""'
    code => '
        value = event.get("someField")
        if value == @lastValue
            event.cancel
        end
        @lastValue = value
    '
}

In either case you will need pipeline.workers set to 1 and pipeline.ordered set to auto (the default in v7.x).

2 Likes