Preventing specific events from being indexed

[tsmori (Tim Mori)]

Is there a way to prevent specific events from being indexed? I don't mean specific fields, but conditionals for a field.

For example, we're ingesting netflow traffic data and I want to exclude all events that have both source and destination IPs from a specific range. While I know I can exclude these from search results, the point would be to cut down on the index sizes because netflow data is massive.

I'm guessing the best way might be with a pipeline processor, but honestly the docs for using pipelines is so arcane, I can't even get one to compile outside of something that's just descriptive. Even if I can get such a pipeline created, I don't know how it would get applied to the existing architecture.

[Badger]

You could do something like

cidr {
    add_field => { "[@metadata][srcMatch]" => true }
    address => [ "%{srcIp}" ]
    network => [ "169.254.0.0/16", "fe80::/64" ]
}
cidr {
    add_field => { "[@metadata][dstMatch]" => true }
    address => [ "%{dstIp}" ]
    network => [ "169.254.0.0/16", "fe80::/64" ]
}
if [@metadata][srcMatch] and [@metadata][dstMatch] { drop {} }

[tsmori (Tim Mori)]

Thanks, I'll give that a shot. I tried something similar with the cidr filter, setting a tag and then trying to drop by matching tags, but while Logstash didn't complain it also didn't work.