Hello All,
So my logstash installation (full elastic stack running on one machine, ubuntu 19) is configured to accept syslogs from our firewall over UDP on port 514. I can use tcpdump to see that there is indeed traffic coming in on port 514. However, Logstash is picking up nothing. I turned on debug-level logging and here is logstash's logs:
[2019-06-12T11:49:54,677][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.0"}
[2019-06-12T11:50:05,081][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-06-12T11:50:05,555][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-12T11:50:07,117][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-12T11:50:07,764][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-06-12T11:50:07,766][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-06-12T11:50:07,787][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-12T11:50:07,796][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-12T11:50:07,814][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-12T11:50:07,837][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-12T11:50:07,850][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-06-12T11:50:07,851][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-06-12T11:50:07,855][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-12T11:50:07,941][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dy$
[2019-06-12T11:50:09,267][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2019-06-12T11:50:15,102][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-06-12T11:50:15,129][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2c3d05cc run>"}
[2019-06-12T11:50:15,528][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"192.168.50.225:514"}
[2019-06-12T11:50:15,673][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-06-12T11:50:15,992][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-12T11:50:16,396][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"192.168.50.225:514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-06-12T11:50:17,428][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
I have looked at many other topics to try to find other solutions but I have found none that work. I used setcap to allow binding to port 514. Here is my logstash config file:
input {
udp {
host => "192.168.50.225"
port => 514
type => "syslog"
}
}
filter { }
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "syslog-%{+YYYY.MM.dd}"
}
}
If I need to provide more information just let me know. Please help.