i found large message in some log and i count word of log result : 2.5k++
and i using this code..
My sample code
</>
filter {
if "service_daily" in [tags] {
mutate {
gsub => [
"message", "TIMESTAMP|", "",
"message", "LOGTYPE|", "",
"message", "CALL_SERVICE|", "",
"message", "THREAD|", "",
"message", "METHOD|", "",
"message", "URI|", "",
"message", "REQID|", "",
"message", "REQHEADERS|", "",
"message", "REQBODY|", "",
"message", "RESPSTATUS|", "",
"message", "RESPTIME|", "",
"message", "RESPBODY|", "",
"message", "ERRORMESSAGE|", "",
"message", "EXCEPTION|", ""
]
}
#Gsub because i want to set format log >> value1|value2|value3|...valueN
grok {
break_on_match => false
match => [ "message","%{GREEDYDATA:timestamp}|%{GREEDYDATA:logtype}|%{GREEDYDATA:call_service}|%{GREEDYDATA:thread}|%{GREEDYDATA:method}|%{GREEDYDATA:uri}|%{GREEDYDATA:reqid}|%{GREEDYDATA:reqheaders}|%{GREEDYDATA:reqbody}|%{GREEDYDATA:respstatus}|%{GREEDYDATA:resptime}|%{GREEDYDATA:respbody}|%{GREEDYDATA:respbody}|%{GREEDYDATA:respbody}|%{GREEDYDATA:errormessage}|%{GREEDYDATA:exception}" ]
}
.....
.....
</>
it's working for this case but another logs filter is not working
if i using this code
grok {
break_on_match => false
match => [ "message","%{GREEDYDATA:timestamp}|%{GREEDYDATA:logtype}|%{GREEDYDATA:call_service}|%{GREEDYDATA:thread}|%{GREEDYDATA:method}|%{GREEDYDATA:uri}|%{GREEDYDATA:reqid}|%{GREEDYDATA:reqheaders}|%{GREEDYDATA:reqbody}|%{GREEDYDATA:respstatus}|%{GREEDYDATA:resptime}|%{GREEDYDATA:respbody}|%{GREEDYDATA:errormessage}|%{GREEDYDATA:exception}" ]
}
When i run this code value of other field include to timestamp field
This is error from code
Invalid format: "2017-10-09T18:15:14.036|[INFO ]....." is malformed at "|[INFO ]....."
How can i handle with this case (Large message in logfile)
Thank you