# Filebeat 'multiline' multiline pattern

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 22, 2018, 11:04am UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030 "2018-02-22T11:04:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![r2cX](https://avatars.discourse-cdn.com/v4/letter/r/bbce88/32.png) [@r2cX](https://discuss.elastic.co/u/r2cX)\
**Post date:** [February 22, 2018, 11:04am UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030/1 "2018-02-22T11:04:32Z")

</div>

Filebeat 6.2.2

Has anyone tried a multiline.pattern that can span 2 lines (e.g. include \n). I have been struggling with this type of log type. This represents a single request-response log. I used to have filebeat send them as 2 separate events and aggregate them in Logstash but it's not very efficient and aggregate timeouts happen on real world logs.

```
2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
GET /fiz/baz HTTP/1.0
host: my.site.com
connection: close
accept: application/json
user-agent: xxx (xxx)
more_key: more_value

2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
2018-02-05T00:00:03.085485Z 1.454ms
HTTP/1.0 200 OK
Content-Type: application/json;charset=UTF-8

{
  "time": "2018-02-04T00:00:00Z",
  "res": {
    "key": "value"
  }
}
2018-02-05T00:01:02.169645Z 1.998ms

```

So in this case, a multiline pattern of ({timestamp} {ip} {user}) will yield two separate events. I have been testing with something similar to ({timestamp} {ip} {user} **\n** {method} {uri} {httpVer}) but filebeat sends the everything in the log file as a single huge event instead. (I can provide the actual regex that matches in regex101.. but it's quite lengthy because of the ipv6)

I have also tried with multiline.flush\_pattern set at ((\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}).(\d{6})Z (\d)+(.(\d)\*)?ms\n\f) (A form feed character comes after each req/response block) but that doesn't work as well.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 3, 2018, 11:53pm UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030/2 "2018-03-03T23:53:48Z")

</div>

For testing complex regexp I can recommed the link here: [https://www.elastic.co/guide/en/beats/filebeat/6.2/\_test\_your\_regexp\_pattern\_for\_multiline.html](https://www.elastic.co/guide/en/beats/filebeat/6.2/_test_your_regexp_pattern_for_multiline.html)

So in the above example you would expect it to be 2 events in the end?

---

<div class="post-metadata">

**Author:** ![r2cX](https://avatars.discourse-cdn.com/v4/letter/r/bbce88/32.png) [@r2cX](https://discuss.elastic.co/u/r2cX)\
**Post date:** [March 5, 2018, 9:33am UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030/3 "2018-03-05T09:33:59Z")

</div>

I want it as one event. But the first line also matches another line - so I try to use an expression that matches 2 lines instead.

I just tried with the Go Playground:

```
var pattern = `^(\d{4})-(\d{2})-(\d{2})T(\d{2})\:(\d{2})\:(\d{2})\.(\d{6})Z ([\d\.\:\w]+) (bob@gmail.com)([\n\r]+)GET`
var negate = false

var content = `2018-02-05T00:00:03.085031Z 0:0:0:0:0:0:0:1 bob@gmail.com
GET /fiz/baz HTTP/1.0
host: my.site.com

```

Adding the newline characters breaks the match.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 12, 2018, 7:05am UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030/4 "2018-03-12T07:05:10Z")

</div>

Haven't dealt with newline chars in regexp yet. I wonder if you need to use some escaping here?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2018, 7:05am UTC](https://discuss.elastic.co/t/filebeat-multiline-multiline-pattern/121030/5 "2018-04-09T07:05:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.