As in Filebeat is sending an empty line?
What if just the source field was empty? As in, I groked a "jsonfieldfoo" field out of a line that is not just json. But that field doesn't always exist. Would grok create an empty "jsonfieldfootarget" in that case?
For example, for gitlab production logs, I do this:
filebeat prospector:
- paths:
- /var/log/gitlab/gitlab-rails/production.log
encoding: plain
document_type: gitlab_production
multiline:
pattern: ^Started
negate: true
match: after
logstash filter:
grok {
match => ["message", "Started\s%{WORD:verb}\s\"%{NOTSPACE:request}\"\sfor\s%{IPORHOST:clientip}\s(at)\s.*(?<timestamp>%{TIMESTAMP_ISO8601}\s[-+]{1}\d{4})((?<main_message>.*)Parameters:\s(?<project_parameters>\{.*\})(?<main_message_two>.*\n)*(Completed?\s?%{NUMBER:response}(?<response_message>.*))?)?"]
add_field => {"combinestuff" => "blah" }
}
mutate {
gsub => ["project_parameters", "=>",":"]
}
{
json {
source => "project_parameters"
target => "project_parameters"
}
project_parameters does not always exist. So would the json filter throw that error if that happened?
I did try wrapping the json filter in if [project_parameters]{ }, but did not stop the error messages from showing up.
Also, just to double check, when Filebeat sends a line to Logstash using the plain encoding, the actual log line is in the message field, correct?