Thanks for your verification, Magnus.
It works now. Only thing I changed was from:
filter {
if [type] == "docker" {
if [message] =~ "\A\{.+\}\z" {
json { source => "message" }
}
# Some non-json messages are now tagged with _jsonparsefailure. We dont care.
#mutate { remove_tag => [ "_jsonparsefailure" ] }
}
}
to:
filter {
if [type] == "docker" {
if [message] =~ "\A\{.+\}\z" {
json {
source => "message"
}
}
# Some non-json messages are now tagged with _jsonparsefailure. We dont care.
#mutate { remove_tag => [ "_jsonparsefailure" ] }
}
}
I don't think the line breaks are the cause for this though. Strange. Anyway, it works. Thank again for the confirmation.
For the sake of completeness:
Non-Json message:
# echo '{ "message": "Server is shutting down. It ran for 71s.\r" }' | /usr/share/logstash/bin/logstash -f /tmp/json-test.conf
13:34:23.841 [[.monitoring-logstash]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2}
13:34:23.846 [[.monitoring-logstash]-pipeline-manager] INFO logstash.pipeline - Pipeline .monitoring-logstash started
13:34:23.858 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
13:34:23.871 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
13:34:23.922 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9601}
{
"@timestamp" => 2017-10-02T11:34:23.888Z,
"@version" => "1",
"host" => "inf-elk01-t",
"message" => "Server is shutting down. It ran for 71s.\r"
}
Json Message:
# echo '{ "message": "{\"pid\":8,\"hostname\":\"2fa0a81cebef\",\"level\":40,\"time\":1506929233374,\"msg\":\"Requiring lib/events is deprecated, use liServer.events instead\",\"v\":1}\r" }' | /usr/share/logstash/bin/logstash -f /tmp/json-test.conf
13:34:42.345 [[.monitoring-logstash]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2}
13:34:42.352 [[.monitoring-logstash]-pipeline-manager] INFO logstash.pipeline - Pipeline .monitoring-logstash started
13:34:42.370 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
13:34:42.385 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
13:34:42.450 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9601}
{
"@timestamp" => 2017-10-02T11:34:42.408Z,
"@version" => "1",
"host" => "inf-elk01-t",
"message" => "{\"pid\":8,\"hostname\":\"2fa0a81cebef\",\"level\":40,\"time\":1506929233374,\"msg\":\"Requiring lib/events is deprecated, use liServer.events instead\",\"v\":1}\r"
}