Hi all!
I hope someone could help me because I dig the entire internet without finding a solution..
Here we are:
I have a filebeat agent running on pfsense 2.3.x ( filebeat version 6.0.0-alpha3-git877f311 (amd64), libbeat 6.0.0-alpha3-git877f311). I use it to manage my snort logs:
cat filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/snort/*/alert
input_type: log
document_type: SnortIDPS
output:
logstash:
enabled: true
timeout: 15
index: filebeat
hosts: ["192.168.4.10:5000"]
So I send my events to a logstash server, but sometimes the logs give me this:
ERR Failed to publish events: write tcp 192.168.4.1:26869->192.168.4.10:5000: write: broken pipe
On the logstash side, here is my configuration:
input {
beats {
type => "beats"
port => 5000
}
}
output {
if [type] == "SnortIDPS" {
elasticsearch {
hosts => localhost
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
The debug logs keep have this message:
[2017-07-07T19:43:23,151][DEBUG][logstash.pipeline ] filter received {"event"=>{"@timestamp"=>2017-07-07T23:43:15.453Z, "offset"=>251146, "@version"=>"1", "beat"=>{"hostname"=>"fw001", "name"=>"fw001", "version"=>"6.0.0-alpha3-git877f311"}, "host"=>"fw001", "prospector"=>{"type"=>""}, "source"=>"/var/log/snort/snort_em1_vlan536159/alert", "message"=>"07/07/17-19:06.712347 ,119,33,1,\"(http_inspect) UNESCAPED SPACE IN HTTP URI\",TCP,192.168.5.20,41590,192.168.4.2,3128,0,Unknown Traffic,3,", "type"=>"beats", "tags"=>["beats_input_codec_plain_applied"]}}
[2017-07-07T19:43:23,157][DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2017-07-07T23:43:15.453Z, "offset"=>251146, "@version"=>"1", "beat"=>{"hostname"=>"fw001", "name"=>"fw001", "version"=>"6.0.0-alpha3-git877f311"}, "host"=>"fw001", "prospector"=>{"type"=>""}, "source"=>"/var/log/snort/snort_em1_vlan536159/alert", "message"=>"07/07/17-19:43:06.712347 ,119,33,1,\"(http_inspect) UNESCAPED SPACE IN HTTP URI\",TCP,192.168.5.20,41590,192.168.4.2,3128,0,Unknown Traffic,3,", "type"=>"beats", "tags"=>["beats_input_codec_plain_applied"]}}
[2017-07-07T19:43:26,941][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-07-07T19:43:31,942][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-07-07T19:43:36,943][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-07-07T19:43:41,944][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-07-07T19:43:46,943][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-07-07T19:43:51,945][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
So, it seems that filebeat is working, logstash reveice the logs but after that never process them and create the index in elasticsearch...
Any ideas?
Thanks!
