Only netflow v5 is shown although both v9 and v5 are collected

[massimo]

Hello,

as the topic says netflow information (v5 and v9) is sent to logstash. Both are processed and stored in elasticsearch as expected. When opening the index in Kibana only v5 logs are shown although there are much less of this version than of v9, a ration of 1:30. Only after defining a filter which excludes v5 logs v9 logs are shown.

Any ideas?

complete stack on version 5.4

Regards,
massimo

[warkolm (Mark Walkom)]

Providing your config would be helpful

[rcowart (Rob Cowart)]

I have been doing A LOT with Flow technologies lately. I have posted files for a basic Netflow v5 and v9 solution on Github. It uses only out-of-the-box components and there is definitely more possible with plugins, but maybe you will find it useful to figure out your issue.

Rob

[massimo]

Robert, I already know your efforts in this area. I am familiar with Netflow and other tools and I've also already tried this setup with version 2.4 which did not show this behaviour. I guess this is anyhow not really related to v5/v9 since I noticed that Kibana shows me for this index and its available fields very few, sometimes only one value for a particular field although there a more distinct values available.

[massimo]

Nothing special in the logstash config, i.e., input is udp with codec netflow. One filter translation for protocol (number) to protocol name conversion and an output to elasticsearch.

[warkolm (Mark Walkom)]

Providing the config would be very helpful

It may look simple, but that's entirely relative and you may have something in there that explains that. If we cannot see it, we cannot be sure.

[massimo]

input {
        udp {
                port => 2055
                tags => ["netflow"]
                codec => netflow {
                        versions => [5,9]
                }
        }
}

filter {
        if "netflow" in [tags] {
                translate {
                        dictionary => [ "1", "icmp",
                                        "6", "tcp",
                                        "17", "udp",
                                        "41", "ipv6",
                                        "47", "gre",
                                        "50", "esp",
                                        "50", "ah",
                                        "89", "ospf"
                                        ]
                        field => "[netflow][protocol]"
                        destination => "[netflow][protocolname]"
                }
                translate {
                        dictionary_path => '/tmp/devices.yaml'
                        field => "host"
                        destination => "hostname"
                }

        }
}

output {
        if "netflow" in [tags] {
                elasticsearch { hosts => localhost
                                index => "netflow-logs"
                }
        }
}

[rcowart (Rob Cowart)]

I tried your config (other than the translate filter). While there are things that can be done to make the data more usable, I can't see anything that looks incorrect based on your config.

Can you share what you are seeing that you feel is wrong?

[massimo]

As I said, the table to the left on Discover only presents v5 as available value for any given timeframe although there are are many more logs for v9.

[warkolm (Mark Walkom)]

Do the fields show in the index management page?

[massimo]

All fields are there but Kibana does not show me "9" as shown in the screenshot above - only "5" is shown. If I exclude "5" using a filter "9" is shown:

[Christian_Dahlqvist]

The values shown in the field list are based on the sample of 500 documents that is retrieved, not the entire data set. This is why you might only be seeing some of the available values at first and then the rest once a subset of documents have been excluded from the data fetched.

[massimo]

This is confusing since there is much more data coming in with "9".

[massimo]

The initial tests when it comes to dealing with netflow in elasticStack have been completed. After disabling the sending of netflow information of various devices I noticed that one of the devices' system time was 2 hours in the future, i.e., although nothing was being sent anymore it still looked like as if data was coming in for another 2 hours. This is the very same device which was sending netflow v5 data - maybe this is related why Kibana was only showing this device's data as mentioned earlier.