Specify multiline prospector and pipeline for elasticsearch output

I am new to filebeat and would like to set up a prospector reading a log file in which one entry is of the following form of a http request and response. It is multiline and i am not sure how to specify, that the [RESPONSE] and [REQUEST] blocks should be part of the same entry.
I also would like to use some pipeline to pass this in json format to elasticsearch output. Which processors should i use?

[REQUEST (webAPIHandlers).Download-fm] [155438493.626241] [2019-04-04 15:35:36 +0200]
GET /minio/download/bucket/lethe1policy.json?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTQzODQ5OTYsInN1YiI6ImFkbWluIn0.3vAMrQbU0MvT3vKoO4m-Mfc3uEyYR7Y7zddE24ogPIY1ZdFKBOVRYKekJCLB7z_132G5kiNpS7wP9ZMc0RhmZw
Host: zon-psr-sub001.zontal.vmserver:9000
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Dnt: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,
Referer: http://zon-psr-sub001.zontal.vmserver:9000/minio/bucket/
Connection: keep-alive
Upgrade-Insecure-Requests: 1

[RESPONSE] [155438493.626241] [2019-04-04 15:35:36 +0200]
200 OK
Cache-Control: no-store
X-Xss-Protection: 1; mode=block
Accept-Ranges: bytes
Last-Modified: Thu, 04 Apr 2019 13:35:01 GMT
Content-Length: 263
Content-Disposition: attachment; filename="lethe1policy.json"
Vary: Origin
Content-Security-Policy: block-all-mixed-content
X-Amz-Request-Id: 159248DD3222DC74
Server: Minio/RELEASE.2019-03-13T21-59-47Z
Etag: "772755e428b7d836c106dbeda54099da"
Content-Type: application/json

Hey @thola
if you follow an example here

you can specify a pattern to be '^\[REQUEST'

so having configuration

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

will include everything which comes after [REQUEST as a single message up to the point where another occurrence of [REQUEST is found

if there are another logs in the mix you can also use multiline.flush_pattern for termination of the line

multiline.flush_pattern: '^Content-Type: '

after occurrence of content type message be considered a whole.
also you may think about max-lines options if this is something deterministic

What i basically want to do is make a json out of the lines with : rest-of-the-line. and have twi different fields for that, one for the request part and one for the response part. Both are part of one log entry

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.