I have a file like this.
[2021-04-22T13:10:06.549Z] Start new event
[2021-04-22T13:10:06.549Z] + set
[2021-04-22T13:10:06.549Z] BUILD_NUMBER=401
[2021-04-22T13:10:06.549Z] BRANCH_NAME=develop
[2021-04-22T13:10:06.549Z] STAGE_NAME='Build Stage'
[2021-04-22T13:10:06.549Z] + ./scripts/make-application.sh
[2021-04-22T13:10:06.549Z] End event
[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event
[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event
[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event
[2021-04-22T13:10:06.549Z] Start new event
[2021-04-22T13:10:06.549Z] + set
[2021-04-22T13:10:06.549Z] BUILD_NUMBER=401
[2021-04-22T13:10:06.549Z] BRANCH_NAME=develop
[2021-04-22T13:10:06.549Z] STAGE_NAME='Test Stage'
[2021-04-22T13:10:06.549Z] + ./scripts/blahblah.sh
[2021-04-22T13:10:06.549Z] End event
and a pattern section in the filebeat.yml file like this
multiline.pattern: 'Start new event'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: 'End event'
and for output I get three event messages
"message": "[2021-04-22T13:10:06.549Z] Start new event\n[2021-04-22T13:10:06.549Z] + set\n[2021-04-22T13:10:06.549Z] BUILD_NUMBER=401\n[2021-04-22T13:10:06.549Z] BRANCH_NAME=develop\n[2021-04-22T13:10:06.549Z] STAGE_NAME='Build Stage'\n[2021-04-22T13:10:06.549Z] + ./scripts/make-application.sh\n[2021-04-22T13:10:06.549Z] End event"
"message": "[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event\n[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event\n[2021-04-22T13:10:06.549Z] Some other line which I want to be a single line event"
"message": "[2021-04-22T13:10:06.549Z] Start new event\n[2021-04-22T13:10:06.549Z] + set\n[2021-04-22T13:10:06.549Z] BUILD_NUMBER=401\n[2021-04-22T13:10:06.549Z] BRANCH_NAME=develop\n[2021-04-22T13:10:06.549Z] STAGE_NAME='Test Stage'\n[2021-04-22T13:10:06.549Z] + ./scripts/blahblah.sh\n[2021-04-22T13:10:06.549Z] End event"
but this is not what I want. I want the first message and the third message concatenated together but the lines in between I want to remain as one line, one event??
I've tried all combinations of match and negate settings but none of them work. The documentation gives an example of concatenating a Java exception so that suggests it should work on files where there are some exceptions in amongst other lines but in my case above it does not work...?
I think I need it to work like this -
If (match start event) then {add lines} until {end event} else {output line}
but can't figure out how to make it do this if it is even possible
Any ideas before I give up?
Cheers