Socket module reports incorrect IP version

[rcowart (Rob Cowart)]

I am posting here first before opening an issue on Github. While conducting some testing on the new TCP socket module I have found that in some situations an incorrect value is reported for system.socket.family. I have observed this with metricbeat versions 5.2.1 and 5.2.2 (the only two that I tested). Notice in the sample data below that the local and remote addresses are clearly IPv4, however the family is designated as ipv6.

NOTE: metricbeat is running on CentOS7 with kernel 3.10.0-514.6.2

{
    "@timestamp"e => e2017-02-20T18:00:56.312Z,
    "system"e => e{
        "socket"e => e{
            "process"e => e{
                "exe"e => ee"/usr/java/jdk1.8.0_102/jre/bin/java"e,
                "cmdline"e => ee"/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms512m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash --path.config /etc/koios-es/logstash/conf.d/flow -r"e,
                "pid"e => ee3163e,
                "command"e => ee"java"e
            },
            "family"e => ee"ipv6"e,
            "remote"e => e{
                "port"e => ee9200e,
                "ip"e => ee"192.168.12.41"e
            },
            "user"e => e{
                "name"e => ee"root"e,
                "id"e => ee0e
            },
            "local"e => e{
                "port"e => ee43288e,
                "ip"e => ee"192.168.12.32"e
            },
            "direction"e => ee"outgoing"e
        }
    },
    "@metadata"e => e{
        "beat"e => ee"metricbeat"e,
        "type"e => ee"metricsets"e
    },
    "beat"e => e{
        "hostname"e => ee"logstash2"e,
        "name"e => ee"logstash2"e,
        "version"e => ee"5.2.1"e
    },
    "@version"e => ee"1"e,
    "host"e => ee"logstash2"e,
    "metricset"e => e{
        "rtt"e => ee15630e,
        "module"e => ee"system"e,
        "name"e => ee"socket"e
    },
    "type"e => ee"metricsets"e,
    "tags"e => e[
    e   [0] ee"beats_input_raw_event"e
    ]
}

Has anyone else noticed this?

I am able to workaround/correct this using the CIDR filter in logstash, however it would be good to see if this can be corrected.

Rob

[andrewkroh]

I see the same behavior for some connections on my machine. The family value is a direct translation from int to string of what the kernel tells metricbeat. This is probably caused by a socket that is both ipv4 and ipv6. It sounds similar to https://github.com/elastic/beats/issues/3306.

I see the same thing from ss. Does ss show ipv4 addresses when you request it to display only IPv6 sockets (-6)?

ss -ta6n
ESTAB      0      2969      ::ffff:10.0.0.1:443   ::ffff:10.0.2.13:33932

[rcowart (Rob Cowart)]

Yeah, I am seeing the same thing with ss that you are.

ESTAB       0      0     ::ffff:192.168.12.32:33254     ::ffff:192.168.12.41:9200               

I tried disabling IPv6 in /etc/sysconfig/network-scripts/ifcfg-ens160, but that made no difference.

I then disabled IPv6 altogether by adding the following to /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1

By disabling IPv6 globally, Metricbeat now reports all of the connections consistently as IPv4. No real surprise... this was afterall the brute force method.

Since this is just the way the kernel reports things, I can live with it. Fortunately I have all of my beats data passing through logstash in order to transform the data to fit my model. "fixing" this in logstash was trivial using the cidr filter to recognize the IP address pattern.

Thanks for taking a look.

Rob