Graphing Netflow Data

Elastic Stack Kibana
1

I am not 100% sure whether my problem is with Graph or Logstash.

I am able to collect Netflow data from Cisco Meraki via Logstash into Elasticsearch with a few warnings from Logstash on flowset_id.

However, when viewed in Kibana I see all the Netflow data is there under _source and no apparent way to make it aggregatable (hence why I wondered if I actually have an issue with Logstash)

Within Graph, I do not see the _source field as a field source for vertices - obviously what I would like to do is be able to pull out IP addresses from within the data.

Have others seen the issue - or got a fix?

If I look at a JSON formatted entry in Kibana I see:

{
  "_index": "logstash-netflow-2017.01.05",
  "_type": "netflow",
  "_id": "AVlukdZAdgIVGljWGndZ",
  "_score": null,
  "_source": {
    "netflow": {
      "in_pkts": 5,
      "ipv4_dst_addr": "46.101.240.99",
      "first_switched": "2017-01-05T12:19:28.999Z",
      "flowset_id": 5206,
      "l4_src_port": 49152,
      "version": 9,
      "flow_seq_num": 19448800,
      "ipv4_src_addr": "172.16.102.180",
      "in_bytes": 4181,
      "protocol": 6,
      "input_snmp": 1,
      "last_switched": "2017-01-05T12:19:29.999Z",
      "out_pkts": 5,
      "out_bytes": 936,
      "l4_dst_port": 31280
    },
    "@timestamp": "2017-01-05T12:19:28.000Z",
    "@version": "1",
    "host": "10.10.101.254",
    "type": "netflow",
    "tags": []
  },
  "fields": {
    "@timestamp": [
      1483618768000
    ]
  },
  "sort": [
    1483618768000
  ]
}

Thanks

2

The answer lies in how this source document is mapped into your index. Can you supply the mapping definition?

3

For the record, here is an example mapping and configuration I have used before to look at IP comms data [1]

When you use Graph to say "summarise my network" there's a lot of conversations to look at so it helps to have some additional indexed fields to help de-duplicate the chatter you choose to sample.
The example below shows how a typical mapping might look and uses an ingest pipeline to add the documents. The graph queries can then use these indexed fields to present useful summaries.
Another tip is to put internal and external IPs into different indexed fields so in the UI you can give each field a different icon or colour for the vertices that are shown.

[1] https://gist.github.com/markharwood/c478ea0192857b9cdb24ad41d4d14fcd

4

Thanks for the pointer - I think there was indeed an index issue in the way I was using elasticsearch.

I recreated a new index with a new name using the template in the Cisco Blog, tweaked Logstash, and then used the new index in Kirbana - which seems to have fixed things.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.