Unable to parse Filebeat using multiline pattern

Elastic Stack Beats
1

Hello, I am having issues trying to parse a multiline file due to the \n characters being read in.

I've spent forever trying to figure out how to break my fields out due to this. Would someone mind giving me a hand?
My pattern worked below when I was testing manually, and of course not putting in the CR's.... well almost worked, I could never get VPDateTime to move into the @timestamp field.

My message looks like this
"""10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:03.021,,start,newcall,
10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:03.021,,start,ani,33333
10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:03.021,,start,areacode,NA
10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:03.021,,start,exchange,NA
10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:08.349,,end,result,normal
10.9.68.10.1485802323021.30.CEngine,01/30/2017 10:52:08.349,,end,duration,500"""

My grok pattern is like below:

"grok": {
          "field": "message",
          "patterns": ["%{IP:clientIP}.%{VPID:VPID}.%{NOTSPACE:VPApp},%{DATESTAMP:VPDateTime},%{GREEDYDATA:message2},,end,duration,%{NUMBER:Duration}"]
          , "pattern_definitions": {
            "VPID": "%{WORD}.%{WORD}",
            "Duration": "end,duration,.%{WORD}"
             }
          },
          "remove": {
            "field": "message2"
          }

Just to add, my multiline pattern (which seems to work) looks like:
type: log
paths:
{{ range $i, $path := .paths }}
- {{path}} {{ end }} exclude_files: [".gz"]

multiline.pattern: '^.*bstart,newcall.*$'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: '.*bend,duration.*$'
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.