Having issue with DNS filter


(David Gomez) #1

I'm using logstash as a NetFlow collector, and it is working well for a start. I'm getting all the flow, and I want to use filters to be able to resolve ips. I create a simple filter (my first one, but it is not working. I'm new with this, so bare with me.

Here is the config file:

input {
    udp {
      host => "0.0.0.0"
      port => 2025
      codec => netflow {
        versions => [9]
      }
    }
  }

  filter {
  dns {
    reverse => [ "ipv4_dst_addr" ]
    action => "replace"
  }
}

  output {
    stdout { codec => rubydebug }
      elasticsearch {
        index => "conn-flow-netflow"
        hosts => "10.50.0.20"
      }
    }

Here is the error message that I'm getting on the logs:

[2018-01-17T14:17:47,338][WARN ][logstash.filters.dns     ] DNS filter could not perform reverse lookup on missing field {:field=>"ipv4_dst_addr"}

I will appreciate if any help.


(Kurt Schraeyen) #2

Looks like the "ipv4_dst_addr" field doesn't exist. What fields do you see in the output when you leave the dns filter out of the config?


(David Gomez) #3

How do I look into that? I see the field on Kibana. Is there a way to see it on logstash?, but I do see the field in Kibana.


(Kurt Schraeyen) #4

according to the config you posted you have the logs output to stdout, so you should see them in your terminal when you start logstash from the command line


(David Gomez) #5

This is what I get on the log:

[2018-01-17T15:14:03,956][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 11, column 1 (byte 130) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:inblock in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:inconverge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`


(Kurt Schraeyen) #6

That looks like an invalid logstash configuration. To see if the netflow codec creates the "ipv4_dst_addr" field it should be as simple as this:

input {
    udp {
      host => "0.0.0.0"
      port => 2025
      codec => netflow {
        versions => [9]
      }
    }
 }

output {
        stdout { codec => rubydebug }
}

(David Gomez) #7

I still see the same output. I wonder where is the rubydebug? because the log that I'm checking is in this file:

/var/log/logstash/logstash-plain.log

(David Gomez) #8

I got it, I just needed to go to the logstash.yml and enable debug from there.

Here is my output:

 [2018-01-17T17:28:47,849][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :in_pkts]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :in_bytes]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :ipv4_src_addr]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :ipv4_dst_addr]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :input_snmp]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :output_snmp]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :last_switched]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :first_switched]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint16, :l4_src_port]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint16, :l4_dst_port]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :src_as]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint32, :dst_as]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:ip4_addr, :bgp_ipv4_next_hop]}
[2018-01-17T17:28:47,850][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :src_mask]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :dst_mask]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :protocol]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :tcp_flags]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :src_tos]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:uint8, :direction]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260     {:field=>[:forwarding_status, :forwarding_status]}
[2018-01-17T17:28:47,851][DEBUG][logstash.codecs.netflow  ] Field definition complete for template 260 {:field=>[:uint16, :flow_sampler_id]}

(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.