Logstash netflow V9

jdambly · February 2, 2016, 9:17pm

I'm getting the following from logstash

Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>936, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1320, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1432, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>388, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>936, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>828, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1432, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1048, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>936, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1376, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>440, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1268, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1100, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1432, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>772, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1100, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>1432, :level=>:warn}
Template length doesn't fit cleanly into flowset {:template_id=>256, :template_length=>47, :record_length=>608, :level=>:warn}

how do I figure out what the issue is here? some of the netflow data is not looking accurate

ssi · February 17, 2016, 8:58am

did you find a solution for this?

marke72 · February 17, 2016, 6:40pm

We just had this issue recently. It came up after a few devices' firmware was updated. The mix of the old firmware sending with the new firmware caused this error to occur. Once we updated the rest of the devices to the same firmware, the errors stopped.
I have also heard that a mix of devices such as Cisco and another device that sends Netflow might cause this to happen. The fix was to have Logstash receive the devices' Netflow on different ports from each other. Is that right, @jystol?

jdambly · February 17, 2016, 7:31pm

I installed logstash using Yum, it turns out the repo does not have the
most up to date code. I manually downloaded the latest RPM of elasticsearch
and logstash, this resolved the issue for me.

--Jeff

ssi · February 17, 2016, 8:03pm

thanks will try that, are you using version9 of netflow, what does your logstash conf and netflow.yaml look like?

jdambly · February 17, 2016, 8:41pm

here is what my config looks like. Since I have a lot of devices I made an
index for each deivce, and day. this lets my delete old indexs to save
space.

input {
udp {
port => 6343
type => "netflow"
codec => netflow {
definitions => "/etc/logstash/codecs/netflow.yaml"
versions => [9]
}
}
}
output {
if [type] == "netflow"{
elasticsearch {
index => "netflow-%{host}-%{+YYYY.MM.dd}"
hosts => ["10.1.1.17:9200"]
}
}
}

pd7sjc-es-05:/root# cat /etc/logstash/codecs/netflow.yaml
1:

4
:in_bytes
2:
4
:in_pkts
3:
4
:flows
4:
:uint8
:protocol
5:
:uint8
:src_tos
6:
:uint8
:tcp_flags
7:
:uint16
:l4_src_port
8:
:ip4_addr
:ipv4_src_addr
9:
:uint8
:src_mask
10:
4
:input_snmp
11:
:uint16
:l4_dst_port
12:
:ip4_addr
:ipv4_dst_addr
13:
:uint8
:dst_mask
14:
4
:output_snmp
15:
:ip4_addr
:ipv4_next_hop
16:
2
:src_as
17:
2
:dst_as
18:
:ip4_addr
:bgp_ipv4_next_hop
19:
4
:mul_dst_pkts
20:
4
:mul_dst_bytes
21:
:uint32
:last_switched
22:
:uint32
:first_switched
23:
4
:out_bytes
24:
4
:out_pkts
25:
:uint16
:min_pkt_length
26:
:uint16
:max_pkt_length
27:
:ip6_addr
:ipv6_src_addr
28:
:ip6_addr
:ipv6_dst_addr
29:
:uint8
:ipv6_src_mask
30:
:uint8
:ipv6_dst_mask
31:
:uint32
:ipv6_flow_label
32:
:uint16
:icmp_type
33:
:uint8
:mul_igmp_type
34:
:uint32
:sampling_interval
35:
:uint8
:sampling_algorithm
36:
:uint16
:flow_active_timeout
37:
:uint16
:flow_inactive_timeout
38:
:uint8
:engine_type
39:
:uint8
:engine_id
40:
4
:total_bytes_exp
41:
4
:total_pkts_exp
42:
4
:total_flows_exp
43:
:skip
44:
:ip4_addr
:ipv4_src_prefix
45:
:ip4_addr
:ipv4_dst_prefix
46:
:uint8
:mpls_top_label_type
47:
:uint32
:mpls_top_label_ip_addr
48:
4
:flow_sampler_id
49:
:uint8
:flow_sampler_mode
50:
:uint32
:flow_sampler_random_interval
51:
:skip
52:
:uint8
:min_ttl
53:
:uint8
:max_ttl
54:
:uint16
:ipv4_ident
55:
:uint8
:dst_tos
56:
:mac_addr
:in_src_max
57:
:mac_addr
:out_dst_max
58:
:uint16
:src_vlan
59:
:uint16
:dst_vlan
60:
:uint8
:ip_protocol_version
61:
:uint8
:direction
62:
:ip6_addr
:ipv6_next_hop
63:
:ip6_addr
:bgp_ipv6_next_hop
64:
:uint32
:ipv6_option_headers
64:
:skip
65:
:skip
66:
:skip
67:
:skip
68:
:skip
69:
:skip
80:
:mac_addr
:in_dst_mac
81:
:mac_addr
:out_src_mac
82:
:string
:if_name
83:
:string
:if_desc
225:
:ip4_addr
:ipv4_src_addr_postnat
226:
:ip4_addr
:ipv4_dst_addr_postnat
227:
:uint16
:l4_src_port_postnat
228:
:uint16
:l4_dst_port_postnat

ssi · February 17, 2016, 8:51pm

thanks!

and this is working with netflow v9 from a cisco device

ssi · February 18, 2016, 8:49am

hi i tried with your config on newest version of logstash. it runs with missing template for id 263&256 for a while and then throws this error

Exception in inputworker {"exception"=>#<BinData::UnRegisteredTypeError: acl_id_asa>, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/registry.rb:41:in lookup'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:19:ininitialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:49:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:84:inadd_field'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/struct.rb:372:in sanitize_fields'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/struct.rb:371:in sanitize_fields'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/struct.rb:340:insanitize_parameters!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:290:in sanitize!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:207:ininitialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/sanitize.rb:189:in sanitize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/base.rb:284:inextract_args'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/base.rb:254:in extract_args'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/base.rb:80:ininitialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/warnings.rb:22:in initialize_with_warning'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.3/lib/logstash/codecs/netflow.rb:147:indecode_netflow9'", "org/jruby/RubyKernel.java:1242:in catch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.3/lib/logstash/codecs/netflow.rb:137:indecode_netflow9'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/array.rb:208:in each'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/array.rb:208:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.3/lib/logstash/codecs/netflow.rb:136:indecode_netflow9'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.3/lib/logstash/codecs/netflow.rb:86:in decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/array.rb:208:ineach'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.2.0/lib/bindata/array.rb:208:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.3/lib/logstash/codecs/netflow.rb:85:in decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:96:ininputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:73:in `udp_listener'"], :level=>:error}
E