Hello, i don't know how to parse this log file:
VERSION: 1.0.2
00:00:01.159[http-thread-pool-8080(5)] - ERROR - SOMETHING #1
00:00:22.372[http-thread-pool-8080(1)] - ERROR - SOMETHING #2
00:00:23.837[http-thread-pool-8080(2)] - ERROR - SOMETHING #3
...
in first line i have application version, so, i need add new field in json to all log rows, like this:
{"time": "00:00:01.159", "thread_name": "[http-thread-pool-8080(5)]", "message": "ERROR - SOMETHING #1", version: "1.0.2"}
{"time": "00:00:22.372", "thread_name": "[http-thread-pool-8080(1)]", "message": "ERROR - SOMETHING #2", version: "1.0.2"}
{"time": "00:00:23.837", "thread_name": "[http-thread-pool-8080(2)]", "message": "ERROR - SOMETHING #3", version: "1.0.2"}
I would just put in a grok filter in your config to match the version line and ignore it using the drop filter.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html
Something like
grok {
match => "^VERSION: 1.0.2"
add_field => [ "[@metadata][ignore]", "true" ]
}
if([@metadata][ignore] == "true") { drop {} }
@aethos, your solution won't work since @Alexey_Plotnikov wants the version string from the first line to propagate to all subsequent events in the same file.
There is unfortunately no standard way of solving this. Sure, you could have a simple filter plugin that remembers field values between events, but at some point Logstash will restart and then the file input won't read the file from the beginning and then there is no version field to remember.
Ah I misread him.
So, i have only one way: parse first line via bash-script(awk, sed, etc.), delete first line and append version to file name?
Yeah, that's one way. You could also append the version string to each line.