So I finally got my config working and now i recieve grokparsefailure, and it is because filebeat is pushing the headers to logstash and it is not configured of course to filter for the headers.
For example, my bro DNS log, the headers look like this:
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2017-09-01-07-45-30
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id rtt query
When filebeat pushes it to logstash, logstash displays grokparsefailure... example:
{"message":"#separator \x09","@version":"1","@timestamp":"2017-09-04T20:00:07.425Z","source":"/nsm/bro/logs/current/conn.log","offset":16,"type":"CONN_LOG","input_type":"log","beat":{"hostname":"secon","version":"5.5.1","name":"secon"},"host":"secon","tags":["beats_input_codec_plain_applied","_grokparsefailure"]}
Then the actual log that tails after the headers and gets parsed correctly, which look like this (which this is not the issue here (it filters fine):
1504251920.517562 CqzFJ64gxBvuYgndF2 192.168.123.105 33217 164.124.101.2 53 udp 12996 - - - - - - 3 NXDOMAIN F F F F 0 - - F
So the question is, how do i drop the headers that contain the #, such as #set_sperator. Would it be the same process as the #fields that also contain the ts, uid etc...
My guess would be:
filter {
if [message] == "#seperator" {
drop { }
}
}
or
filter {
if [message] == "#fields" {
drop { }
}
}
Would the later drop everything in the #fields record, or do i have to write out everything in the header line?