UDP listener problem

Elastic Stack Logstash
1

Hey i have a probleme with UDP:

my conf : firewalld and selinux disable

logtstash 5.5-2

my conf :

input {
udp {
port => 9556
host => "1.1.1.55"
workers => 2
codec => netflow {
versions => [5, 9]
target => "nf"
}
type => "netflow"
}`

logstash.yml :

#
path.data: /var/lib/logstash
#
pipeline.workers: 2
#
# How many workers should be used per output plugin instance
#
pipeline.output.workers: 1
#
pipeline.batch.size: 125
#
#
pipeline.batch.delay: 5
#
#he pipeline configuration for the main pipeline
#
path.config: /etc/logstash/conf.d/*
#
#
#
# log.level: info
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins:

my error : 2017-08-29T16:57:27,428][INFO ][logstash.pipeline ] Pipeline main started [2017-08-29T16:57:27,429][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"1.1.1.55:9556"} [2017-08-29T16:57:27,445][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"1.1.1.55:9556", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} [2017-08-29T16:57:27,452][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2017-08-29T17:01:30,404][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [2017-08-29T17:01:41,465][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent. [2017-08-29T17:01:41,473][WARN ][logstash.agent ] stopping pipeline {:id=>"main"} [2017-08-29T17:01:41,832][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3705:in select'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:93:in udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:56:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:456:in inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:449:in start_input'"]}`

help me plz

2

From the message:
[logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK.

logstash is executed in 'test configuration mode'? In that case it is normal that it exits.

3

i run this command :

/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/0001-input-netflow.conf --path.settings /etc/logstash/

and after i restart logstash and i have :

[2017-08-29T17:38:14,833][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-08-29T17:39:49,357][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[2017-08-29T18:01:57,325][WARN ][logstash.runner          ] SIGTERM received. Shutting down the agent.
[2017-08-29T18:01:57,340][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-08-29T18:01:57,450][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3705:in `select'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:93:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:56:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:456:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:449:in `start_input'"]}
4

remove the '-t'

Anyways... i could not reproduce the messages that you get when closing down Logstash. :thinking:

5

when i stop logstash and start logstash ( systemctl start logstash.service)

i have :

[2017-08-29T18:22:24,674][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-08-29T18:22:24,676][INFO ][logstash.pipeline ] Pipeline main started
[2017-08-29T18:22:24,678][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"1.1.1.55:9556"}
[2017-08-29T18:22:24,692][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-08-29T18:22:24,695][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"1.1.1.55:9556", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-08-29T18:30:17,968][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-08-29T18:30:17,977][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-08-29T18:30:18,315][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<IOError: closed
stream>, :backtrace=>["org/jruby/RubyIO.java:3705:in `select'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:93:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.1/lib/logstash/inputs/udp.rb:56:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:456:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:449:in `start_input'"]}

6

now i have this :

[2017-08-29T20:11:40,130][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2017-08-29T20:11:40,132][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-08-29T20:11:40,196][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-08-29T20:11:40,198][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-08-29T20:11:40,232][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-08-29T20:11:40,235][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["//localhost:9200"]}
[2017-08-29T20:11:40,250][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-08-29T20:11:40,252][INFO ][logstash.pipeline ] Pipeline main started
[2017-08-29T20:11:40,253][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"1.1.1.55:9556"}
[2017-08-29T20:11:40,268][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"1.1.1.55:9556", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-08-29T20:11:40,279][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

netstat -aux
Connexions Internet actives (serveurs et établies)
Proto Recv-Q Send-Q Adresse locale Adresse distante Etat
udp6 0 0 1.1.1.55:9556 [::]:*

and with tcpdump i can see the netflow

but i have nothing

7

Do you mean that you see the traffic arriving into that host/port?

Test first with an console output instead of Elasticsearch (or in addition to) to see if at least it is processed.
Actually you could also check the logstash API endpoint to see if there are events processed

8

In my output i have in a file and ES but i am nothing

9

i think there is a bug with UDP / netflow :

In all case i receive the flow

first :
when my elk server can't communicate with my network equipment but i can see the netflow in it network interface with tcpdump, i have no data processed

second case :

when my elk server can communicate with my network equipment and i can see the netflow in it network interface with tcpdump, i have data processed

it does not mean anything, with UDP there is not communication as in TCP

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.