I have a apache tomcat log of the following format:
Mar 15, 2014 8:18:33 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 576 ms
ERROR: org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:200)
Mar 15, 2014 8:18:33 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 576 ms
ERROR: org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:200)
Now using multiline, i am able to extract two events from the above logs. However since there are multiple log level messages(INFO and ERROR) in a single event, i am unable to tag fields to to log messages.
Current grok filter cons looks like :
filter {
multiline {
patterns_dir => "/etc/logstash/patterns"
#pattern => "[%{TIMESTAMP_ISO8601}"
pattern => "^%{TIMESTAMP_CUSTOM}"
negate => "true"
what => "previous"
}
if [type] == "tomcat-log" and "INFO" in [message] {
mutate {
add_tag => [ "INFO_TAG"]
}
grok {
patterns_dir => "/etc/logstash/patterns"
pattern => "%{TIMESTAMP_CUSTOM:date}"
}
grok {
patterns_dir => "/etc/logstash/patterns"
break_on_match => false
match => { "message" => "%{CUSTOM_WARNINGLEVEL:warningLevel} %{LINE:level_info}( %{CUSTOM_TRACE_ERROR:error} )?(%{CUSTOM_TRACE_EXCEPTION:exception})?(%{CUSTOM_TRACE_OTHERS:exception_info})?(%{CUSTOM_TRACE_CAUSED_BY:caused_by})?(%{CUSTOM_TRACE_OTHERS:cause_info})?(%{GREEDYDATA:messageText})?" }
}
}
}
The current output looks like :
{
"warningLevel": [
[
"INFO:"
]
],
"level_info": [
[
"Initialization processed in 576 ms\nERROR: org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during processing of message.\norg.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?\nat org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:200)"
]
],
"error": [
[
null
]
],
"exception": [
[
null
]
],
"exception_info": [
[
null
]
],
"caused_by": [
[
null
]
],
"cause_info": [
[
null
]
],
"messageText": [
[
""
]
]
}
Is there a way i can extract and tag ERROR: log message in the example above?