Help with grok filter [ipv4]

[raged]

Afternoon all,

We are utilizing metricbeat to send host info by using the metadata that is described in this article. It eventually gives me a field that is populated with IPs from all of the network adapters:

"host": {
  "geo": {
    "name": "nyc-dc1-rack1",
    "region_iso_code": "NY",
    "country_iso_code": "US",
    "continent_name": "North America",
    "city_name": "New York",
    "location": "40.7128, -74.0060",
    "region_name": "New York"
  },
  "name": "lulherro",
  "ip": [
    "fe80::d55c:7b6:e4a9:75ce",
    "10.3.87.127",
    "fe80::2827:c72f:c34:a5fb",
    "169.254.165.251",
    "fe80::55bc:c197:969c:d531",
    "169.254.213.49",
    "fe80::75:adc8:a95b:a3f8",
    "169.254.163.248",
    "fe80::5072:8bd2:f703:4252",
    "169.254.66.82"
  ],

My end goal is to parse out a valid IPV4 (non 169) so I can end up doing a subnet query to get client's location based on subnet.

I am trying to parse out a valid IP from this array:

fe80::d55c:7b6:e4a9:75ce, 10.3.87.127, fe80::2827:c72f:c34:a5fb, 169.254.165.251, fe80::55bc:c197:969c:d531, 169.254.213.49, fe80::75:adc8:a95b:a3f8, 169.254.163.248, fe80::5072:8bd2:f703:4252, 169.254.66.82
169.254.66.82
169.254.66.82
169.254.66.82169.254.66.82
169.254.66.82

If I use %{IPV4:validIP} as the grok filter using https://grokdebug.herokuapp.com/:

However, when I use logstash like so:

if [host][ip] {
    grok {
        match => {"[host][ip]" => '%{IPV4:validIP}'}
    }
}

I get data showing up like this:

"validIP": [
  "10.3.87.127",
  "169.254.165.251",
  "169.254.213.49",
  "169.254.163.248",
  "169.254.66.82"
],

How do I get logstash to only match non 169 values like it did on the grok debug site above?

Thanks,
Raged

[Badger]

You could use mutate+join to create a single string, then pick the first V4 IP address from it using grok. If the one you want is not the first then use a negative lookahead assertion in the pattern to reject any addresses starting with '169.'

Another option would be to split the validIP field, so that you have multiple events, then drop {} any that have IPs you do not want.

The grok debugger is iterating over the array of messages that you want to test against the pattern. grok does not do that.

[raged]

Thanks for the suggestion's @Badger.

For anyone else wanting to do this, this is what I ended up with:

if [host][ip] {
    grok {
        match => {"[host][ip]" => '%{IPV4:validIP}'}
  
    }
ruby { code => "
        array = event.get('validIP').each {|item|
        if item =~ /^169.*/
          
        else
          event.set('validIP',item)
        end
        }
"
}
} 

It essentially takes a field from this:

  "ip": [
    "fe80::d55c:7b6:e4a9:75ce",
    "10.3.87.127",
    "fe80::2827:c72f:c34:a5fb",
    "169.254.165.251",
    "fe80::55bc:c197:969c:d531",
    "169.254.213.49",
    "fe80::75:adc8:a95b:a3f8",
    "169.254.163.248",
    "fe80::5072:8bd2:f703:4252",
    "169.254.66.82"
  ],

to this:

"validIP": "10.3.87.127",