Here is my input log file names as : cadence-cdslmd-dlhl0939_dlhl0940_dlhl0941-5.11.2019-event.log
15:56:24 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
15:56:32 (cdslmd) OUT: "Analog_Design_Environment_XL" gautamd@dlhsx00005
15:56:33 (lmgrd) TIMESTAMP 5/13/2019
15:56:35 (cdslmd) OUT: "OASIS_Simulation_Interface" mnarasim@dlhsx00012
15:56:37 (cdslmd) IN: "111" guptasri@dlhsx00010
15:56:42 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
My logstash filter
input {
beats {
port => 5044
}
}
filter {
grok {
match => [ "message", "%{DATA:event_timestamp} \(%{DATA:lmgrd}\) TIMESTAMP %{DATE:monthday}" ]
tag_on_failure => [ "message_data" ]
}
mutate {
add_field => { "taskId" => "all" }
}
if "message_data" not in [tags] {
aggregate {
task_id => "%{taskId}"
code => "map['monthday'] = event.get('monthday')"
}
}
else {
aggregate {
task_id => "%{taskId}"
code => "if (map['monthday'] != nil) then event.set('monthday', map['monthday']) else event.set('monthday', (event.get('source').split('-')[3])) end"
}
grok {
match => [ "message", "%{TIME:event_timestamp} \(%{DATA:lic_vendor_name}\) (?<event_type>(OUT|IN|DENIED))\: \"%{DATA:lic_feature_name}\" %{DATA:user_name}@%{HOSTNAME:host_name}" ] }
mutate { replace => ["event_timestamp", "%{monthday} %{event_timestamp}" ] }
date { match => [ "event_timestamp", "M/d/yyyy HH:mm:ss","M.d.yyyy HH:mm:ss","ISO8601" ] target => "event_timestamp" }
}
}
output {
stdout {
codec => rubydebug
}
}
Before input line 3, the monthday should come from filename( 3rd field :5.11.2019) and after that monthday should get from the logfile itself from TIMESTAMP line ( 5/13/2019)
The output coming corresponding to input lines are
Line 1 : "event_timestamp" => 2019-05-11T10:26:24.000Z, "monthday" => "5.11.2019"
Line 2: "event_timestamp" => 2019-05-11T10:26:32.000Z "monthday" => "5.11.2019"
Line3 : "event_timestamp" => "15:56:33" "monthday" => "5/13/2019"
Line4 : "event_timestamp" => 2019-05-13T10:26:35.000Z "monthday" => "5/13/2019"
Line 5: "event_timestamp" => 2019-05-11T10:26:37.000Z "monthday" => "5.11.2019"
Line 6: "event_timestamp" => 2019-05-11T10:26:42.000Z, "monthday" => "5.11.2019"
Expected output should be
Line 1 : "event_timestamp" => 2019-05-11T10:26:24.000Z, "monthday" => "5.11.2019"
Line 2: "event_timestamp" => 2019-05-11T10:26:32.000Z "monthday" => "5.11.2019"
Line3 : "event_timestamp" => "15:56:33" "monthday" => "5/13/2019"
Line4 : "event_timestamp" => 2019-05-13T10:26:35.000Z "monthday" => "5/13/2019"
Line 5: "event_timestamp" => 2019-05-11T10:26:37.000Z "monthday" => "5/13/2019"
Line 6: "event_timestamp" => 2019-05-11T10:26:42.000Z, "monthday" => "5/13/2019"
Please suggest where I am doing it wrong?