How can I visualize "top-talkers" from netflow data?

[sliddjur]

Hello I want to visualize top-talkers from netflow data.

With a top-talkers I mean data sent from source to destination, as well as the data replied from the same dst and src.

I would like to stack this data in a vertical bar, and list the top-talkers based on flows/bytes etc. Something similar to this is what I'm trying to achive http://blog.linoxide.com/wp-content/uploads/2013/11/ntopng_flows.png

[spalger (Spencer Alger)]

Can you give me an example of what your documents might look like? I'm thinking something like this:

{
  "src": "1.2.3.4",
  "dest": "2.3.4.5",
  "bytes": 42
}

[sliddjur]

@spalger yes thats what it looks like.
I'm intrested in netflow.ipv4_src_addr, netflow.ipv4_dst_addr and netflow.in_bytes

{
  "_index": "netflow-2016.12.23",
  "_type": "netflow",
  "_id": "AVkq4HlGRFxRys8jOYJN",
  "_score": null,
  "_source": {
    "netflow": {
      "ipv4_src_host": "192.168.0.51",
      "in_pkts": 3,
      "first_switched": "2016-12-23T08:47:52.999Z",
      "flowset_id": 256,
      "l4_src_port": 56054,
      "ipv4_next_hop": "x.x.128.1",
      "ipv4_dst_host": "edge-z-m-mini-shv-01-amt2.facebook.com",
      "in_bytes": 156,
      "protocol": 6,
      "tcp_flags": 17,
      "xlate_src_addr_ipv4": "192.168.0.51",
      "l4_dst_port": 443,
      "output_snmp": 15,
      "out_src_mac": "e4:8d:8c:20:ed:e2",
      "dst_mask": 0,
      "xlate_src_port": 0,
      "ipv4_dst_addr": "31.13.64.36",
      "src_tos": 0,
      "in_dst_mac": "xx:xx:xx:20:ed:e6",
      "src_mask": 0,
      "xlate_dst_port": 0,
      "version": 9,
      "flow_seq_num": 29029,
      "ipv4_src_addr": "192.168.0.51",
      "last_switched": "2016-12-23T08:48:04.999Z",
      "input_snmp": 13,
      "xlate_dst_addr_ipv4": "31.13.64.36",
      "protocol_name": "TCP"
    },
    "dst_geoip": {
      "timezone": "Europe/Dublin",
      "ip": "31.13.64.36",
      "latitude": 53.3472,
      "country_code2": "IE",
      "country_name": "Ireland",
      "continent_code": "EU",
      "country_code3": "IE",
      "location": [
        -6.2439,
        53.3472
      ],
      "longitude": -6.2439
    },
    "@timestamp": "2016-12-23T08:48:20.000Z",
    "@version": "1",
    "host": "192.168.0.1",
    "src_geoip": {},
    "type": "netflow",
    "tags": [
      "netflow",
      "_geoip_lookup_failure"
    ]
  },
  "fields": {
    "netflow.first_switched": [
      1482482872999
    ],
    "netflow.last_switched": [
      1482482884999
    ],
    "@timestamp": [
      1482482900000
    ]
  },
  "sort": [
    1482482900000
  ]
}

[ppisljar (Peter Pisljar)]

Its not possible to do it with scripted fields as ip fields are not accessible from scripts. I suppose you want to do this without reindexing your data ? @spalger did you have something specific in mind ?

[sliddjur]

@ppisljar is it possible with strings? How?

[Christian_Dahlqvist]

Since you want to group records with different source and destination fields, I suspect the most efficient way would be to create some type of conversation ID at indexing time. Add the source and destination fields (host and port*) to an array, sort it and concatenate it to a string with some kind of separator.

If you also want to show duration like in your example at scale, you may want to consider creating an entity-centric index for that.

[ppisljar (Peter Pisljar)]

yes, with strings its possible. but if you can update your mappings (reindex, or your data is not yet indexed) then what Christian suggests would be best.

with scripted fields you could do something like:

return (doc['ip_from_string'].value < doc['ip_to_string'].value ? doc['ip_from_string'].value + '-' + doc['ip_to_string'].value : doc['ip_to_string'].value + '-'+ doc['ip_from_string'].value;

this will return something in form 127.0.0.1-200.12.123.2 for this scripted fields (where the smaller ip always come first to make sure we don't have two pairs 127.0.0.1-200.12.123.2 and 200-12.123.2-127.0.0.1 (we want to treat them as one talker group)

let me say this again, if you can add another field at the index time, that would be much better.