Filebeat 'multiline' multiline pattern

[r2cX]

Filebeat 6.2.2

Has anyone tried a multiline.pattern that can span 2 lines (e.g. include \n). I have been struggling with this type of log type. This represents a single request-response log. I used to have filebeat send them as 2 separate events and aggregate them in Logstash but it's not very efficient and aggregate timeouts happen on real world logs.

2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
GET /fiz/baz HTTP/1.0
host: my.site.com
connection: close
accept: application/json
user-agent: xxx (xxx)
more_key: more_value


2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
2018-02-05T00:00:03.085485Z 1.454ms
HTTP/1.0 200 OK
Content-Type: application/json;charset=UTF-8

{
  "time": "2018-02-04T00:00:00Z",
  "res": {
    "key": "value"
  }
}
2018-02-05T00:01:02.169645Z 1.998ms

So in this case, a multiline pattern of ({timestamp} {ip} {user}) will yield two separate events. I have been testing with something similar to ({timestamp} {ip} {user}\n{method} {uri} {httpVer}) but filebeat sends the everything in the log file as a single huge event instead. (I can provide the actual regex that matches in regex101.. but it's quite lengthy because of the ipv6)

I have also tried with multiline.flush_pattern set at ((\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}).(\d{6})Z (\d)+(.(\d)*)?ms\n\f) (A form feed character comes after each req/response block) but that doesn't work as well.

[ruflin]

For testing complex regexp I can recommed the link here: https://www.elastic.co/guide/en/beats/filebeat/6.2/_test_your_regexp_pattern_for_multiline.html

So in the above example you would expect it to be 2 events in the end?

[r2cX]

I want it as one event. But the first line also matches another line - so I try to use an expression that matches 2 lines instead.

I just tried with the Go Playground:

var pattern = `^(\d{4})-(\d{2})-(\d{2})T(\d{2})\:(\d{2})\:(\d{2})\.(\d{6})Z ([\d\.\:\w]+) (bob@gmail.com)([\n\r]+)GET`
var negate = false

var content = `2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
GET /fiz/baz HTTP/1.0
host: my.site.com

Adding the newline characters breaks the match.

[ruflin]

Haven't dealt with newline chars in regexp yet. I wonder if you need to use some escaping here?