Specify multiline prospector and pipeline for elasticsearch output

Elastic Stack Beats
1

I am new to filebeat and would like to set up a prospector reading a log file in which one entry is of the following form of a http request and response. It is multiline and i am not sure how to specify, that the [RESPONSE] and [REQUEST] blocks should be part of the same entry.
I also would like to use some pipeline to pass this in json format to elasticsearch output. Which processors should i use?

[REQUEST (webAPIHandlers).Download-fm] [155438493.626241] [2019-04-04 15:35:36 +0200]
GET /minio/download/bucket/lethe1policy.json?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTQzODQ5OTYsInN1YiI6ImFkbWluIn0.3vAMrQbU0MvT3vKoO4m-Mfc3uEyYR7Y7zddE24ogPIY1ZdFKBOVRYKekJCLB7z_132G5kiNpS7wP9ZMc0RhmZw
Host: zon-psr-sub001.zontal.vmserver:9000
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Dnt: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8
Referer: http://zon-psr-sub001.zontal.vmserver:9000/minio/bucket/
Connection: keep-alive
Upgrade-Insecure-Requests: 1

[RESPONSE] [155438493.626241] [2019-04-04 15:35:36 +0200]
200 OK
Cache-Control: no-store
X-Xss-Protection: 1; mode=block
Accept-Ranges: bytes
Last-Modified: Thu, 04 Apr 2019 13:35:01 GMT
Content-Length: 263
Content-Disposition: attachment; filename="lethe1policy.json"
Vary: Origin
Content-Security-Policy: block-all-mixed-content
X-Amz-Request-Id: 159248DD3222DC74
Server: Minio/RELEASE.2019-03-13T21-59-47Z
Etag: "772755e428b7d836c106dbeda54099da"
Content-Type: application/json

2

Hey @thola
if you follow an example here

you can specify a pattern to be '^\[REQUEST'

so having configuration

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

will include everything which comes after [REQUEST as a single message up to the point where another occurrence of [REQUEST is found

if there are another logs in the mix you can also use multiline.flush_pattern for termination of the line
including

multiline.flush_pattern: '^Content-Type: '

after occurrence of content type message be considered a whole.
also you may think about max-lines options if this is something deterministic

3

What i basically want to do is make a json out of the lines with : rest-of-the-line. and have twi different fields for that, one for the request part and one for the response part. Both are part of one log entry