Hi,
I have configure the filebeat to tomcat log and below is my filebeat multiline configuration:
multiline.pattern: "^(\s+|\t)|(Caused by:)|(Exception:)|(System Error:)"
multiline.negate: false
multiline.match: after
multiline.flush_pattern: "^((?!\t).)"
So here is the below output, when i pasted the below to the tomcat log at same time:
18:18:24.891 QL t@-1858021632 End Command
18:18:24.891 VERB t@-1858021632 input params: (session=EDF3EA6DABE00B000C67E987ECF84A82.v6dp1-01:mx810075420afc940d6:(ApproveRejectProcess_jsp.java:383)), type=Java, msg=CLASS LOADER Exception: System Error: #5000001: Error: #1900068: java.lang.NullPointerException
Exception: java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
18:18:24.891 VERB t@-1858021632 input params: cmd=unset env USER, allowMultipleOverride=0
OUTPUT:
{
"host" => "localhost.localdomain",
"@version" => "1",
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure"
],
"message" => "18:18:24.891 QL t@-1858021632 End Command\n18:18:24.891 VERB t@-1858021632 input params: (session=EDF3EA6DABE00B000C67E987ECF84A82.v6dp1-01:mx810075420afc940d6:(ApproveRejectProcess_jsp.java:383)), type=Java, msg=CLASS LOADER Exception: System Error: #5000001: Error: #1900068: java.lang.NullPointerException\nException: java.lang.NullPointerException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)",
"@timestamp" => 2018-03-30T14:42:55.405Z,
"source" => "/Tomcat Logs/output.txt",
"offset" => 656,
"beat" => {
"name" => "localhost.localdomain",
"version" => "6.2.3",
"hostname" => "localhost.localdomain"
}
}
If you see the message, it combine all the things i pasted. Rather than to combine only the exception part. I checked the same in https://play.golang.org/p/uAd5XHxscu and below is the output.
matches line
false 18:18:24.891 QL t@-1858021632 End Command
true 18:18:24.891 VERB t@-1858021632 input params: (session=EDF3EA6DABE00B000C67E987ECF84A82.v6dp1-01:mx810075420afc940d6:(ApproveRejectProcess_jsp.java:383)), type=java, msg=CLASS LOADER Exception: System Error: #5000001: Error: #1900068: java.lang.NullPointerException
true Exception: java.lang.NullPointerException
true at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
true at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
true at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
true at java.lang.reflect.Method.invoke(Method.java:606)
false 18:18:24.891 VERB t@-1858021632 input params: cmd=unset env USER, allowMultipleOverride=0
false
So here i except, all the true line should come to output as one event. And all the false one will come in individual event to the output.
But actually, the ouput combines all. Exception (those are marked true in above) and all the false on single output event.
Please let me know, how to fix this. Or please let me know, how to configure the filebeat to consider each entry to log file as individual event unless and until it matches the multiline regex.
Thank You.