Having issue with DNS filter

Elastic Stack Logstash
1

I'm using logstash as a NetFlow collector, and it is working well for a start. I'm getting all the flow, and I want to use filters to be able to resolve ips. I create a simple filter (my first one, but it is not working. I'm new with this, so bare with me.

Here is the config file:

input {
    udp {
      host => "0.0.0.0"
      port => 2025
      codec => netflow {
        versions => [9]
      }
    }
  }

  filter {
  dns {
    reverse => [ "ipv4_dst_addr" ]
    action => "replace"
  }
}

  output {
    stdout { codec => rubydebug }
      elasticsearch {
        index => "conn-flow-netflow"
        hosts => "10.50.0.20"
      }
    }

Here is the error message that I'm getting on the logs:

[2018-01-17T14:17:47,338][WARN ][logstash.filters.dns     ] DNS filter could not perform reverse lookup on missing field {:field=>"ipv4_dst_addr"}

I will appreciate if any help.

2

Looks like the "ipv4_dst_addr" field doesn't exist. What fields do you see in the output when you leave the dns filter out of the config?

3

How do I look into that? I see the field on Kibana. Is there a way to see it on logstash?, but I do see the field in Kibana.

4

according to the config you posted you have the logs output to stdout, so you should see them in your terminal when you start logstash from the command line

5

This is what I get on the log:

[2018-01-17T15:14:03,956][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 11, column 1 (byte 130) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:inblock in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:inconverge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`

6

That looks like an invalid logstash configuration. To see if the netflow codec creates the "ipv4_dst_addr" field it should be as simple as this:

input {
    udp {
      host => "0.0.0.0"
      port => 2025
      codec => netflow {
        versions => [9]
      }
    }
 }

output {
        stdout { codec => rubydebug }
}
7

I still see the same output. I wonder where is the rubydebug? because the log that I'm checking is in this file:

/var/log/logstash/logstash-plain.log
8

I got it, I just needed to go to the logstash.yml and enable debug from there.

Here is my output:

 [2018-01-17T17:28:47,849][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :in_pkts]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :in_bytes]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :ipv4_src_addr]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :ipv4_dst_addr]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :input_snmp]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :output_snmp]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :last_switched]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :first_switched]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint16, :l4_src_port]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint16, :l4_dst_port]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :src_as]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :dst_as]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :bgp_ipv4_next_hop]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :src_mask]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :dst_mask]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :protocol]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :tcp_flags]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :src_tos]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :direction]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:forwarding_status, :forwarding_status]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260 {:field=>[:uint16, :flow_sampler_id]}
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.