5.6.1 Netflow not working


(Sjaak) #1

Hi,

I've updated from 5.4 to 5.6.1 so I could use the latest netflow codec with support for Fortigate devices but now I got no netflow data coming in anymore.

I did not change the config and logstash log shows no errors.

config

input {
   udp {
     port => 9995
	type => "netflow"
	codec => netflow {
}
}
}

filter {
  mutate {
    add_field => {
      "[netflow][ipv4_dst_host]" => "%{[netflow][ipv4_dst_addr]}"
      "[netflow][ipv4_src_host]" => "%{[netflow][ipv4_src_addr]}"
    }
  }

  if ([netflow][l4_dst_port]) {
   mutate {
     add_field => {
      "[netflow][l4_dst_port_translation]" => "%{[netflow][l4_dst_port]}"
     }
   } 
  }

  translate {
    dictionary_path => '/etc/logstash/port_translation.yaml'
    field => "[netflow][l4_dst_port_translation]"
    override => true
    destination => "[netflow][l4_dst_port_translation]"
  }

  dns {
    action => 'replace'
    reverse => "[netflow][ipv4_dst_host]"
  }

  dns {
  action => 'replace'
  reverse => "[netflow][ipv4_src_host]"
  }
}

output {
if [type] == "netflow" {
elasticsearch {
hosts => localhost
user => logstash_internal
password => logstash
index => "netflow-%{+YYYY.MM.dd}"
}
}
}

Logstash log

[2017-09-27T13:24:17,284][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@localhost:9200/"}
[2017-09-27T13:24:17,288][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-09-27T13:24:17,295][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-09-27T13:24:17,304][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2017-09-27T13:24:17,379][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-09-27T13:24:17,387][INFO ][logstash.pipeline        ] Pipeline main started
[2017-09-27T13:24:17,389][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:9994"}
[2017-09-27T13:24:17,397][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:9995"}
[2017-09-27T13:24:17,412][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:9995", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-09-27T13:24:17,413][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:9994", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-09-27T13:24:17,442][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Elastic log doesn't show anything either.

This looks suspiciously similar to the KV filter issue I reported in a previous topic (unresolved) where for whatever reason everything is running but nothing gets into elastic and no index or mapping appears to be created.

Any way to fix this?


Netflow v9 not being recieved over UDP
5.6 fundamentally broken
(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.