Passing a Log4J2 file with multilines


(Whgibbo) #1

Hi,
I've just started playing with Logstash 2.0.0 and so far it looks great :smile:

And it is working fine for single line entries, but not for multiline entries. :frowning:

Here is my config file to pass the file.

input {
file {
path => "F:/test.log"
start_position => beginning
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
filter {
grok {
patterns_dir => "./patterns"
match => {
# - %{MY_LOGMESSAGE:logmessage}
"message" => ["%{TIMESTAMP_ISO8601:timestamp} [%{MY_THREAD_NAME:thread}] %{MY_LOGLEVEL:loglevel} %{JAVACLASS:class} - %{MY_LOGMESSAGE:logmessage}"]
}
}
}
output {
stdout { codec => json }
}

Here is the pattern file that I'm currently using:

MY_THREAD_NAME ([^]]+)
MY_LOGLEVEL ([Tt]race|TRACE|[Dd]ebug|DEBUG|[Ii]nfo|INFO|[Ww]arn|WARN|[Ee]rror|ERROR|)\s*
MY_CLASSNAME (?:[a-zA-Z0-9[]/-]+.)+[A-Za-z0-9[]$]+
MY_LOGMESSAGE ((.|\s)*)

I've been using https://regex101.com to ensure that I've got my pattern correct which is as follows:

(?>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])T:(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?) [([^]]+)] (([Tt]race|TRACE|[Dd]ebug|DEBUG|[Ii]nfo|INFO|[Ww]arn|WARN|[Ee]rror|ERROR|)\s*) ((?:[a-zA-Z0-9[]/-]+.)+[A-Za-z0-9[]$]+) - ((.|\s)*)

Here is a sample of a single line entry:

2015-11-18T09:41:10,436 [pool-4-thread-1 ] INFO com.test.Application - Starting Application

This is the json that is printed to the screen (with some formatting applied):

{
"@timestamp": "2015-11-18T09:41:10.628Z",
"message": "2015-11-18T09:41:10,436 [pool-4-thread-1 ] INFO com.test.Application - Starting Application\r",
"@version": "1",
"host": "localhost",
"path": "F:/test.log",
"timestamp": "2015-11-18T09:41:10,436",
"thread": "pool-4-thread-1 ",
"loglevel": "INFO ",
"class": "com.test.Application",
"logmessage": "Starting Application\r"
}


(Whgibbo) #2

and here is a sample of a multiline entry:

2015-11-18T09:46:56,615 [http-nio-8082-exec-2 ] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Test exception] with root cause
java.lang.IllegalArgumentException: Test exception
at com.test.RandomNumberController(RandomNumberController.java:31) ~[main/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]

Had to truncate due to the number of characters allowed to post. That begs a question, is there a limit to a the length of the line that logstash can process ?


(Whgibbo) #3

When it passes a mulitline the following is returned (had to truncate the message due to the size, but did contain all the multilines):

{
"@timestamp": "2015-11-18T09:46:57.196Z",
"message": "2015-11-18T09:46:56,615 [http-nio-8082-exec-2 ] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Test exception] with root cause\r\njava.lang.IllegalArgumentException: Test exception\n\tat com.test.RandomNumberController.getRandomNumber(RandomNumberController.java:31) ~[main/:?]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]\n\tat sun.reflect.NativeMeth odAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]",
"@version": "1",
"tags": ["multiline","_grokparsefailure"],
"host": "localhost",
"path": "F:/test.log"
}

Anybody have any thoughts ?

Thanks


(Whgibbo) #4

Ok think I resolved it at least with Logstash 2.0. So thought I'd post here so that I don't forget and that it might help somebody else.

Had to modify my Log4J2 pattern slightly..

Pattern: "%d{ISO8601} [%t] %level %logger{36} - %msg%n"

And changed my grok pattern as follows:

"message" => ["%{TIMESTAMP_ISO8601:logtimestamp}%{SPACE}[%{MY_THREAD_NAME:logthread}]%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{MY_CLASSNAME:logclass}%{SPACE}-%{SPACE}%{GREEDYDATA:logmessage}"]

Then changed one of the patterns to:

MY_CLASSNAME (?:[a-zA-Z0-9[]/-]+.)+[A-Za-z0-9[]/$]+


(system) #5