Thanks for the suggestions @Pierhugues! Looks like my YAML config file was not right. I started over using filebeat.yml.rpmnew as my template. With the increased logging level, I am able to see the multiline feature running
2018-12-19T10:30:51.115-0600 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2018-12-19T16:30:51.114Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.3"
},
"offset": 21691179,
"log": {
"flags": [
"multiline"
]
},
However now I've created a new issue for myself. In the logs of the Logstash server I am seeing this error
Dec 19 11:02:31 logstash logstash: [2018-12-19T11:02:31,619][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.12.19", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x7ee1d5d1>], :response=>{"index"=>{"_index"=>"logstash-2018.12.19", "_type"=>"doc", "_id"=>"TQxrx2cB0IW8JrxHNsSD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1581"}}}}}
Should I create a new topic in the Logstash section or do you have any suggestions on what I might try?
I've tried your suggestion here - Logstash errors after upgrading to filebeat-6.3.0 but can't seem to get it to work with my if condition filter in Logstash.
Here's what I have so far (config pasted below has generic servername for beat.hostname for security reasons). Any thoughts? Thanks!
filter {
if [beat.hostname] == "insertservername" {
mutate {
remove_field => [ "[host]" ]
}
mutate {
add_field => {
"host" => "%{[beat][hostname]}"
}
}
}
}