How do I get the raw message of the netflow/ipfix codec?

Elastic Stack Logstash
1

Im using the netflow codec and I seems it doesn't include a raw message field when it outputs to es.

for input I just do this:
udp {
type => "ipfix"
port => "123"
codec => netflow {
versions => [10]
target => ipfix
}
}

Now I want to map geoip field but I dont know what the right field name is:
if [type] == "ipfix" {
clone {
add_field => { "message" => "my_raw_message" }
}
geoip {
source => "sourceIPv4Address"
target => "geoip_src"
}
geoip {
source => "destinationIPv4Address"
target => "geoip_dst"
}
}

The field that I see in es is called sourceIPv4Address, but when trying to use that field name with logstash it doesn't work. I also tried cloning the message to a new field to keep it (assuming the codec even has a "message" field) but that didn't work.

2

There isn't any single-field raw message in ipfix/netflow messages.
The original fields are nested under "ipfix" field (the target value of the input), so the appropriate field for sourceIPv4Address would be "[ipfix][sourceIPv4Address]" or similar.

2 Likes
3

Ah ha! I guess I should have known looking at the events which has this value nested under "ipfix". Guess I didn't put 2 and 2 together. [ipfix][sourceIPv4Address] works for me.

But i'm still not getting the raw message, even doing this:
clone {
add_field => { "ipfix" => "my_raw_ipfix" }
}

Sorry this isn't directly related to my original problem, but one more thing, I seem to be getting _geoip_lookup_failure when processing private IPs- obviously there can't be geoip data for them. Just want to be sure I can leave it as is or if I should be handling the parse errors more gracefully?

THANKS!

4

That is because the ipfix field has nested fields under it (it's essentially a hash) so it can't be copied to another field, not by default means anyway to my knowledge.

As for the geoip failure, I guess you can check whether the IP is private or not before processing the geolocation fields if you want to avoid the errors (it should also slightly boost performance depending on the amount of private IPs since you skip them).

Example regex:

if "[ipfix][sourceIPv4Address]" =~ "(^1((0)|(92\.168)|(72\.((1[6-9])|(2[0-9])|(3[0-1])))|(27))\.)" {
    # geo stuff
}

Other than that, you can either delete that tag or just disregard it altogether if it doesn't mess with the event's presentation. It's far from a critical issue.

5

You can use the cidr filter to determine whether the IP is private or public. An example of how to do this is in my ElastiFlow repo here...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.