5.6.1 Netflow not working

Elastic Stack Logstash
1

Hi,

I've updated from 5.4 to 5.6.1 so I could use the latest netflow codec with support for Fortigate devices but now I got no netflow data coming in anymore.

I did not change the config and logstash log shows no errors.

config

input {
   udp {
     port => 9995
	type => "netflow"
	codec => netflow {
}
}
}

filter {
  mutate {
    add_field => {
      "[netflow][ipv4_dst_host]" => "%{[netflow][ipv4_dst_addr]}"
      "[netflow][ipv4_src_host]" => "%{[netflow][ipv4_src_addr]}"
    }
  }

  if ([netflow][l4_dst_port]) {
   mutate {
     add_field => {
      "[netflow][l4_dst_port_translation]" => "%{[netflow][l4_dst_port]}"
     }
   } 
  }

  translate {
    dictionary_path => '/etc/logstash/port_translation.yaml'
    field => "[netflow][l4_dst_port_translation]"
    override => true
    destination => "[netflow][l4_dst_port_translation]"
  }

  dns {
    action => 'replace'
    reverse => "[netflow][ipv4_dst_host]"
  }

  dns {
  action => 'replace'
  reverse => "[netflow][ipv4_src_host]"
  }
}

output {
if [type] == "netflow" {
elasticsearch {
hosts => localhost
user => logstash_internal
password => logstash
index => "netflow-%{+YYYY.MM.dd}"
}
}
}

Logstash log

[2017-09-27T13:24:17,284][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@localhost:9200/"}
[2017-09-27T13:24:17,288][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-09-27T13:24:17,295][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-09-27T13:24:17,304][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2017-09-27T13:24:17,379][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-09-27T13:24:17,387][INFO ][logstash.pipeline        ] Pipeline main started
[2017-09-27T13:24:17,389][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:9994"}
[2017-09-27T13:24:17,397][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:9995"}
[2017-09-27T13:24:17,412][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:9995", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-09-27T13:24:17,413][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:9994", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-09-27T13:24:17,442][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Elastic log doesn't show anything either.

This looks suspiciously similar to the KV filter issue I reported in a previous topic (unresolved) where for whatever reason everything is running but nothing gets into elastic and no index or mapping appears to be created.

Any way to fix this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.