Multiline codec in Logstash 6.6 not working

Elastic Stack Logstash
1

Hi guys,

I am new with the elk stack, but am trying to implement it to get nicer logs for my application.

Here is my config:

input {
	file {
		path => "/var/log/file.log"
		
		codec => multiline {
			#pattern => "^\d\d\d\d\/\d\d\/\d\d\s\d\d\:\d\d\:\d\d"
			#pattern => "^\d\d\d\d(\/|-)\d\d(\/|-)\d\d(\s|T)\d\d\:\d\d\:\d\d\.\d\d\d"
			#pattern => "^\s{24}.*$"
			#pattern => "^.*(NFO).*$"
			#pattern => "^.*application:\s{15,}.*$"
			#pattern => "^.*application:\s{15,}.*$"
			pattern => "^\w\w\w\s\d\d\s\d\d\:\d\d\:\d\d.*\|.*$"
			negate => "true"
			what => "previous"
		}
	}
}
filter {
if [programname] == "application" {
		dissect {
			mapping => {
				"message" => "%{priority}|%{tag}|%{new_message}"
			}
			add_field => ["log", "application"]
			remove_field => ["message"]
		}
		
		mutate {
			gsub => ["priority","^$","NULL"]
			gsub => ["tag","^$","NULL"]
			gsub => ["new_message","^$","NULL"]
			
			strip => ["priority","tag"]
		}
	}
}
output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "127.0.0.1:9200" ]
    }
    stdout { codec => rubydebug }
  }
}

And here are my sample logs:

Mar 13 13:40:02 hostname application: NFO|HTTPpoll7 |HTTP PACKET FROM 127.0.0.1:port TO 127.0.0.1:port
Mar 13 13:40:02 hostname application: GET /status?module=CNX&mode=extended HTTP/1.1
Mar 13 13:40:02 hostname application: Host: 127.0.0.1:port
Mar 13 13:40:02 hostname application: User-Agent: Zend_Http_Client
Mar 13 13:40:02 hostname application: NFO|HTTP |HTTP PACKET TO 127.0.0.1:port FROM 127.0.0.1:port
Mar 13 13:40:02 hostname application: HTTP/1.0 200 OK
Mar 13 13:40:02 hostname application: Date: Wed, 13 Mar 2019 12:40:02 GMT
Mar 13 13:40:02 hostname application: content-length: 2049
Mar 13 13:40:02 hostname application: content-type: text/plain; charset=UTF-8
Mar 13 13:40:02 hostname application: NFO|HTTP |End of HTTP connection 127.0.0.1:port

As you can see, I've tried multiple regex for the multiline codec, but nothing seems to work, meaning that the output is the same as the input in Kibana. Or maybe I don't how/where to look for the output.

Please help with any advice.

Thanks a lot!

2

why do you need multiline and your log look like each message look like single line.

3

There was some trimming when I added the logs. Here is a snip of how they look like:

logs
logs.PNG927×246 8.2 KB

Ideally, would be something like:

Mar 13 13:40:02 hostname application: NFO|HTTPpoll7 |HTTP PACKET FROM 127.0.0.1:port TO 127.0.0.1:port \n GET /status?module=CNX&mode=extended HTTP/1.1 \n Host: 127.0.0.1:port \n User-Agent: Zend_Http_Client
Mar 13 13:40:02 hostname application: NFO|HTTP      |HTTP PACKET TO 127.0.0.1:port FROM 127.0.0.1:port \n HTTP/1.0 200 OK \n Date: Wed, 13 Mar 2019 12:40:02 GMT \n content-length: 2049 \n content-type: text/plain; charset=UTF-8
Mar 13 13:40:02 hostname application: NFO|HTTP      |End of HTTP connection 127.0.0.1:port

Also, I think I need to mention that my "message" field is " NFO|HTTP |End of HTTP connection 127.0.0.1:port" - as an example.
From my point of view, this would be the ideal outcome, but I am open to suggestions.

4

Try this you will get output
codec => multiline {
pattern => "[a-zA-Z0-9: ]+NFO"
negate => "true"
what => "previous" }

5

Thank you for the suggestion!

I tried it, but it is the same.
Maybe some more details will help: from the above logs, "Mar 13 13:40:02" gets into the timestamp field, "hostname" gets into the sysloghost field, "application" goes into the programname field and the message field starts from ":". They get automatically mapped like this and I only need to use the multiline codec on the message field. Is this possible?

6 
codec => multiline { pattern => " NFO\|" what => "previous" negate => true auto_flush_interval => 2 }

Once you have gotten the date, hostname, application you can then remove them using

mutate { gsub => [ "message", "^[A-Za-z]{3}\s[0-9]{1,2} [0-9]{2}:[0-9]{2}:[0-9]{2} [a-zA-Z]+ [a-zA-Z]+: ", "" ] }

Which will leave you with

       "message" => "NFO|HTTP |HTTP PACKET TO 127.0.0.1:port FROM 127.0.0.1:port\nHTTP/1.0 200 OK\nDate: Wed, 13 Mar 2019 12:40:02 GMT\ncontent-length: 2049\ncontent-type: text/plain; charset=UTF-8"
7

Thank you for the suggestion!

I managed to get the most I could with the aggregate filter.

Thank you all for your input!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.