So with the help of Badger, I got a working grok filter.
Now I need to change it so that it aggregates the lines into one event.
One event looks like this
2019-01-12 16:25:01.889 0EF4 [BERLIN_PSL -] Starting to send messages on output channel 1001, queue 7
2019-01-12 16:25:01.889 0EF4 [BERLIN_PSL -] Call DialogServer.SendMessages
2019-01-12 16:25:02.221 0EF4 [BERLIN_PSL -] End DialogServer.SendMessages last 0 seconds
2019-01-12 16:25:02.222 0EF4 [BERLIN_PSL -] Sent 1 messages in transaction 1 on output channel 1001, queue 7
2019-01-12 16:25:02.222 0EF4 [BERLIN_PSL -] Call DialogServer.SendMessages
2019-01-12 16:25:02.237 0EF4 [BERLIN_PSL -] End DialogServer.SendMessages last 0 seconds
2019-01-12 16:25:02.238 0EF4 [BERLIN_PSL -] Completed sending messages on output channel 1001, queue 7
So my filter looks like this:
grok {
match => {
"message" => [ "^ %{NOTSPACE:date}%{SPACE}%{NOTSPACE:clock} %{WORD:taskid} \[%{WORD:instance} -] %{WORD:status} %{GREEDYDATA:restOfLine}"]
}
}
grok {
match => {
"restOfLine" => [
"to send %{GREEDYDATA}",
"%{INT:batchsent} messages %{GREEDYDATA}",
"DialogServer.SendMessages last %{INT:batchduration} seconds$",
"sending %{GREEDYDATA}"
]
}
}
This takes the four lines that I'm interested in, and places values in fields
Then I've added these lines to get the lines aggregated:
Startup, setting the two counters to 0:
if [status] == "Starting" {
aggregate {
task_id => "%{taskid}"
code => "map['sent'] = 0 ; map['duration'] = 0"
map_action => "create"
}
}
This accumulates the duration (in theory)
if [status] == "End" {
aggregate {
task_id => "%{taskid}"
code => "map['duration'] += event.get('batchduration')"
map_action => "update"
}
}
This accumulates the sent lines (in theory)
if [status] == "Sent" {
aggregate {
task_id => "%{taskid}"
code => "map['sent'] += event.get('batchsent')"
map_action => "update"
}
}
And this should then generate an event with the above accumulated fields.
if [status] == "Completed" {
aggregate {
task_id => "%{taskid}"
code => "event.set('duration', map['duration']) ; event.set('sent', map['sent'])"
end_of_task => true
timeout => 1000
}
}
But if look at the logstash debug I get a few errors about the variable type:
[2019-01-15T13:00:47,802][ERROR][logstash.filters.aggregate] Aggregate exception occurred {:error=>#<TypeError: String can't be coerced into Fixnum>, :code=>"map['duration'] += event.get('batchduration')", :map=>{"sent"=>0, "duration"=>0}, :event_data=>{"date"=>"2019-01-12", "restOfLine"=>"DialogServer.SendMessages last 0 seconds"...
[2019-01-15T13:00:47,805][ERROR][logstash.filters.aggregate] Aggregate exception occurred {:error=>#<TypeError: String can't be coerced into Fixnum>, :code=>"map['sent'] += event.get('batchsent')", :map=>{"sent"=>0, "duration"=>0}, :event_data=>{"date"=>"2019-01-12", "restOfLine"=>"1 messages in transaction 1 on output channel 1001, queue 7", ...
[2019-01-15T13:00:47,809][ERROR][logstash.filters.aggregate] Aggregate exception occurred {:error=>#<TypeError: String can't be coerced into Fixnum>, :code=>"map['duration'] += event.get('batchduration')", :map=>{"sent"=>0, "duration"=>0}, :event_data=>{"date"=>"2019-01-12", "restOfLine"=>"DialogServer.SendMessages last 0 seconds"...
Now, how do I make it so that it understands that it is numbers, and not a string?