I'm experimenting with logstash to collect the output of various updates of things (for example, letsencrypt daily certificate renewals) on multiple servers using the http input. These are almost all multiple line text in the http body ending up in the message field. The whole message is being recorded, except that the newlines have been removed. This appears to have been before the filter stage - I suspect the default "plain" codec used by the http input is doing it. (The newlines have definitely been removed - at first I thought it was kibana simply not displaying \n as
in the html; and I also tried various filters, none of which did anything, suggesting there was no newline to, for example, substitute by that stage).
What can I do to preserve the newlines all the way through the pipeline to make the output readable?
I tried adding a multiline codec with a non-matchable pattern, both with negate true and false, but in all cases no event was then emitted at all. I also tried a plain codec explicitly, but that also dropped the event.
This is v5.5 (es, logstash and kibana). I can't get 6.3 to run on the Raspberry Pi this is using, probably because it is rather memory limited, but 5.5 works ok.
Here's the config (the grok matches seem to work ok):
input {
http {
host => "localhost" # default: 0.0.0.0
port => 8101 # default: 8080
}
}
filter {
grok {
match => { "[headers][request_uri]" => "/(?<system>[^/\?]+)/(?<service>[^/\?]+)(/(?<client>\
[^/\?]+))?" }
}
grok {
match => { "message" => "[^A-Za-z0-9](?<problem>fail|failed|failure|error|Fail|Failed|Failu\
re|Error|FAIL|FAILED|FAILURE|ERROR)(\s+|\:)" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}