DNS Filter And GeoIP can't see ip field

Elastic Stack Logstash
1

Hello,
I am trying to parse netflow logs and would like to perform a DNS lookup and get GeoIP information for the source and destination IPs. However, I can't seem to get Logstash to see the fields that contain the IPs.

This is my Logstash configuration.

input {
        udp {
                port                    => 2055
                codec                   => netflow
                receive_buffer_bytes    => 16777216
                workers                 => 16
        }
}

filter {
        dns {
                reverse => ["destinationIPv4Address", "sourceIPv4Address"]
                action => "append"
        }
        geoip {
                source => "sourceIPv4Address"
                target => "source_geoip"
        }
        geoip {
                source => "destinationIPv4Address"
                target => "destination_geoip"
        }
        translate {
                field => "netflow.destinationTransportPort"
                destination => "netflow.protocol"
                dictionary => {
                        "20"    => "FTP"
                        "21"    => "FTP"
                        "22"    => "SSH"
                        "23"    => "Telnet"
                        "25"    => "SMTP"
                        "53"    => "DNS"
                        "67"    => "DHCP"
                        "68"    => "DHCP"
                        "69"    => "TFTP"
                        "80"    => "HTTP"
                        "110"   => "POP"
                        "123"   => "NTP"
                        "137"   => "NetBIOS"
                        "138"   => "NetBIOS"
                        "139"   => "NetBIOS"
                        "143"   => "IMAP"
                        "161"   => "SNMP"
                        "162"   => "SNMP"
                        "179"   => "BGP"
                        "389"   => "LDAP"
                        "443"   => "HTTPS"
                        "636"   => "LDAPS"
                        "989"   => "FTP over TLS/SSL"
                        "990"   => "FTP over TLS/SSL"
                }
        }
}

output {
        stdout {
                codec   => rubydebug
        }
        elasticsearch {
                hosts   => ["10.0.3.50:9200"]
                index   => "netflow-test"
        }
}

This is the error.

[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"destinationIPv4Address"}
[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"sourceIPv4Address"}
[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"destinationIPv4Address"}
[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"sourceIPv4Address"}
[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"destinationIPv4Address"}
[WARN ] 2018-08-02 09:39:48.269 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"sourceIPv4Address"}
[WARN ] 2018-08-02 09:39:48.270 [Ruby-0-Thread-6@[main]>worker0: :1] dns - DNS filter could not perform reverse lookup on missing field {:field=>"destinationIPv4Address"}

This is the rubydebug stdout for a single log. I am getting v10 (IPFIX) netflow logs.

{
    "@timestamp" => 2018-08-02T06:39:48.000Z,
          "tags" => [
        [0] "_geoip_lookup_failure"
    ],
      "@version" => "1",
          "host" => "10.0.254.1",
       "netflow" => {
                      "flowStartSysUpTime" => 4253399794,
                    "postSourceMacAddress" => "64:d1:54:93:07:b4",
                       "sourceIPv4Address" => "13.107.3.128",
           "postNATDestinationIPv4Address" => "10.0.254.62",
             "destinationIPv4PrefixLength" => 0,
                        "ingressInterface" => 6,
                            "icmpCodeIPv4" => 0,
        "postNAPTDestinationTransportPort" => 0,
                          "tcpControlBits" => 16,
                        "packetDeltaCount" => 1,
                "destinationTransportPort" => 64763,
                      "protocolIdentifier" => 6,
                   "destinationMacAddress" => "64:d1:54:93:07:b8",
                         "octetDeltaCount" => 52,
                  "destinationIPv4Address" => "178.251.40.146",
                    "ipNextHopIPv4Address" => "10.0.254.62",
                  "sourceIPv4PrefixLength" => 0,
                           "ipTotalLength" => 52,
                       "tcpSequenceNumber" => 2120365753,
                        "ipClassOfService" => 0,
                                "igmpType" => 0,
                            "icmpTypeIPv4" => 0,
             "postNAPTSourceTransportPort" => 0,
                               "ipVersion" => 4,
                             "isMulticast" => 0,
                         "egressInterface" => 12,
                        "udpMessageLength" => 0,
                          "ipHeaderLength" => 5,
                                   "ipTTL" => 246,
                "tcpAcknowledgementNumber" => 3296226417,
                        "flowEndSysUpTime" => 4253399794,
                "postNATSourceIPv4Address" => "13.107.3.128",
                     "sourceTransportPort" => 443,
                           "tcpWindowSize" => 65283,
                                 "version" => 10
    }
}

I also tried netflow.destinationIPv4Address and netflow.sourceIPv4Address for the field names but I still got the same error. How can I get Logstash to recognize the IP fields?

2

Use [netflow][destinationIPv4Address]. See https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

1 Like
3

Thank you for the quick reply Magnus. It works just as I want now. However, it still seems to add a [0] "_geoip_lookup_failure" tag. But that doesn't seem to be important since it retrieves the GeoIP information anyway.
Thanks again.

4

I don't want to distract you from building this on your own. However if you want to get up and running very quickly you may want to consider ElastiFlow (https://github.com/robcowart/elastiflow).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.