Filebeat 6.2.2
Has anyone tried a multiline.pattern that can span 2 lines (e.g. include \n). I have been struggling with this type of log type. This represents a single request-response log. I used to have filebeat send them as 2 separate events and aggregate them in Logstash but it's not very efficient and aggregate timeouts happen on real world logs.
2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
GET /fiz/baz HTTP/1.0
host: my.site.com
connection: close
accept: application/json
user-agent: xxx (xxx)
more_key: more_value
2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
2018-02-05T00:00:03.085485Z 1.454ms
HTTP/1.0 200 OK
Content-Type: application/json;charset=UTF-8
{
"time": "2018-02-04T00:00:00Z",
"res": {
"key": "value"
}
}
2018-02-05T00:01:02.169645Z 1.998ms
So in this case, a multiline pattern of ({timestamp} {ip} {user}) will yield two separate events. I have been testing with something similar to ({timestamp} {ip} {user}\n{method} {uri} {httpVer}) but filebeat sends the everything in the log file as a single huge event instead. (I can provide the actual regex that matches in regex101.. but it's quite lengthy because of the ipv6)
I have also tried with multiline.flush_pattern set at ((\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}).(\d{6})Z (\d)+(.(\d)*)?ms\n\f) (A form feed character comes after each req/response block) but that doesn't work as well.