There's no "OUT_BYTES" field when I parse netflow through logstash

f26227279 · January 31, 2018, 9:35am

As title, when I use logstash to parse netflow and ingest flow data into elasticsearch, I found the data in ES is lack of "OUT_BYTES" field.
this is the data in ES:

{
  "_index": "netflow-%{year}-%{month}-%{day}",
  "_type": "doc",
  "_id": "QnjmS2EBwY8ORgd4n7yl",
  "_version": 1,
  "_score": null,
  "_source": {
    "netflow": {
      "in_pkts": 13,
      "version": 5,
      "src_tos": 0,
      "engine_type": 1,
      "ipv4_next_hop": "120.127.163.193",
      "dst_as": 0,
      "dst_mask": 0,
      "sampling_algorithm": 0,
      "src_mask": 0,
      "flow_records": 24,
      "output_snmp": 0,
      "last_switched": "2018-01-31T10:58:40.971Z",
      "first_switched": "2018-01-31T10:58:40.971Z",
      "ipv4_dst_addr": "120.127.163.193",
      "tcp_flags": 0,
      "ipv4_src_addr": "216.58.200.234",
      "protocol": 6,
      "src_as": 0,
      "input_snmp": 191,
      "l4_dst_port": 10029,
      "sampling_interval": 0,
      "l4_src_port": 443,
      "in_bytes": 5548,
      "flow_seq_num": 1259678413,
      "engine_id": 2
    },
    "@timestamp": "2018-01-31T10:58:44.972Z",
    "host": "120.127.163.4",
    "@version": "1"
  },
  "fields": {
    "netflow.first_switched": [
      "2018-01-31T10:58:40.971Z"
    ],
    "@timestamp": [
      "2018-01-31T10:58:44.972Z"
    ],
    "netflow.last_switched": [
      "2018-01-31T10:58:40.971Z"
    ]
  },
  "sort": [
    1517396324972
  ]
}

logstash config:

input{
	udp{
		host => "120.127.XXX.XX"
		port => 5556
		codec => netflow	
	}
}
	filter{
		
	}
output{
elasticsearch {
                hosts => ["120.127.XXX.XX:9200"]
				index => "netflow-%{year}-%{month}-%{day}"
			}
	
		stdout{codec=> rubydebug}
	
}

thank you in advance

jpcarey · January 31, 2018, 11:58am

Why not use the netflow module? https://www.elastic.co/guide/en/logstash/6.1/netflow-module.html

I’m not an expert on netflow, but I’d say there is a little extra processing that is done just by looking at the configuration that the module uses (plus you get the kibana dashboards)

github.com

elastic/logstash/blob/master/modules/netflow/configuration/logstash/netflow.conf.erb

input {
    udp {
        type => "netflow"
        port => <%= setting("var.input.udp.port", 2055) %>
        codec => netflow {
            versions => [5,9]
        }
        workers => <%= setting("var.input.udp.workers", 2) %>
        receive_buffer_bytes => <%= setting("var.input.udp.receive_buffer_bytes", 212992) %>
        queue_size => <%= setting("var.input.udp.queue_size", 2000) %>
    }
}

filter {
    #--------------------
    # The field names for Netflow v5 and v9 differ. We need to normalize to a
    # common data model so that we can work with the resulting events in a common manner.
    #--------------------

    # Process Netflow v5 events.

This file has been truncated. show original

rcowart · January 31, 2018, 5:15pm

You are sending Netflow version 5 flows. Netflow v5 supports ONLY ingress flows, so there is no out_bytes. If you want out_bytes (or an equivalent field) you will need to send Netflow v9 and enable egress flows on the device sending the data.

BTW, a better alternative to the Logstash Netflow module is here...

The Logstash Netflow module was based on v1.0.0 of ElastiFlow and is now quite a bit behind.

system · February 28, 2018, 5:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.