I want to create a grok filter based on the block of multiline log file as given below. Sometimes the log file will not have the "sent VerifyRes" due to transaction failure, so I want to create a grok filter to capture the block of text in the message which has "Request received" and "sent VerifyRes" and also would like to calculate the time taken to complete this transaction (time between beginning and end of transaction)
--
Mon Aug 29 23:45:58.712 2016 INFO: pid 2662 tid 1401: 0 Request received from Machine 127.0.0.1
..
Mon Aug 29 23:45:58.719 2016 INFO: pid 2662 tid 1401: 0 sent VerifyRes
I have written the below grok filter, but it doesn't seem to work fine.
input {
file {
path => "/opt/logstash/trans_log.txt"
start_position => "beginning"
codec => multiline {
pattern => "((?m)%{GREEDYDATA:date} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{YEAR:year} %{LOGLEVEL:loglevel}: pid %{GREEDYDATA:pid} tid %{GREEDYDATA:tid}: 0 Request received from Machine %{IP})|((?m)%{GREEDYDATA:date} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{YEAR:year} %{LOGLEVEL:loglevel}: pid %{GREEDYDATA:pid} tid %{GREEDYDATA:tid}: 0 sent VerifyRes)"
negate => true
what => previous
}
}
}
filter {
grok {
match => { "message" => [
"%{GREEDYDATA:date} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{YEAR:year} %{LOGLEVEL:loglevel}: pid %{GREEDYDATA:pid} tid %{GREEDYDATA:tid}: 0 Request received from Machine %{IP}"
]
remove_tag => [ "_grokparsefailure" ]
}
overwrite => [ "message" ]
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "vereq-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
Any help on this.