Requirement: Depending on the value of a particular field in the last line of the input file, add_field to all the matched pattern outputs with the value of the field.
In the last line of the file, I have a string like "unique_pattern: ".
Currently I'm trying a filter like below,
filter {
grok {
## MATCH ONLY THE LAST LINE
match => { "message" => "unique_pattern: %{DATA:pattern}"
## NOW MATCH OTHER LINES FOR MY REQUIRED PATTERN
match => { "message" => "\[%{TIMESTAMP_ISO8601:log_timestamp}\] \[%{WORD:log_level}] %{GREEDYDATA:log_msg} " }
}
mutate { remove_field => [ "@version", "@timestamp", "type", "host", "path", "locale", "message" ] }
mutate { add_field => { "unique_pattern" => "%{pattern}" } }
}
I'm not sure how to check only the LAST-LINE of the input file and also that at the beginning. And then add the unique_pattern value to each matching line with the value of 'pattern'.
My input file is,
[2020-09-15T07:25:15.298+00:00] [INFO] Log Message1
[2020-09-15T07:25:15.300+00:00] [INFO] Log message2
unique_pattern: World
I want the unique_pattern value 'World' printed in each output.
My current Output,
{"log_level":"INFO","log_msg":"Log","log_timestamp":"2020-09-15T07:25:15.298+00:00","unique_pattern":"%{pattern}","tags":["bipublisher"]}
{"log_level":"INFO","log_msg":"Log","log_timestamp":"2020-09-15T07:25:15.300+00:00","unique_pattern":"%{pattern}","tags":["bipublisher"]}
{"pattern":"World\r","unique_pattern":"World\r","tags":["multiline","bipublisher"]}
My Expected output,
{"log_level":"INFO","log_msg":"Log","log_timestamp":"2020-09-15T07:25:15.298+00:00","unique_pattern":"World\r","tags":["infile"]}
{"log_level":"INFO","log_msg":"Log","log_timestamp":"2020-09-15T07:25:15.300+00:00","unique_pattern":"World\r","tags":["infile"]}
{"pattern":"World\r","unique_pattern":"World\r","tags":["multiline","bipublisher"]}