Filebeat's Multiline not grouping properly

[kellen.hopwood]

Hi All, I am using filebeats to try and pull from a text file that contains http logs, the logs are generally not uniform, so I used powershell to create an end of file marker. However when I then try and send the logs into elastic and later kibana, I get one or two massive logs, not the individual components.

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - 'C:\Program Files\filebeat\logfile.txt'
  multiline:
     pattern: 'XXXXX' 
     negate: true
     match: after
output.logstash:
  hosts: ["localhost:5044"]

The data looks like this.

GET https://scontent-iad3-1.xx.fbcdn.net/v/t39.2093- 
6/36882171_273786243171121_8594480363911249920_n.srt? 
_nc_cat=0&oh=6dae5ff3122dd34ae1cb4dfccfda7779&oe=5C307E15 HTTP/1.1
Host: scontent-iad3-1.xx.fbcdn.net
Connection: keep-alive
Origin: https://www.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/69.0.3497.92 Safari/537.36
Accept: */*
Referer: https://www.facebook.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

AAAAA
XXXXX
HTTP/1.1 200 OK
Last-Modified: Tue, 10 Jul 2018 23:39:25 GMT
Content-Type: text/srt
Timing-Allow-Origin: *
Access-Control-Allow-Origin: *
Expires: Tue, 18 Sep 2018 23:54:05 GMT
Cache-Control: max-age=1209600, no-transform
Date: Sat, 15 Sep 2018 15:14:56 GMT
Access-Control-Expose-Headers: X-FB-CEC-Video-Limit
Connection: keep-alive
Content-Length: 46280
AAAAA
XXXXX

My original attempts to have it sort the data have all gone, rather abysmally, I was hoping someone here could give me some quick pointers to get more acclimated. Thanks!

[jsoriano (Jaime Soriano)]

Hi @kellen.hopwood and welcome

Your configuration looks correct for the example data you posted. What I see that could be wrong is the indentation of the configuration, enabled, paths and multiline must be at the same level as type, i.e.

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - 'C:\Program Files\filebeat\logfile.txt'
  multiline:
    pattern: 'XXXXX' 
    negate: true
    match: after

Could you review the indentation of your configuration file?

[kellen.hopwood]

Hi @jsoriano, I checked the indentation, and it was incorrect in my post, I corrected it, but it looks like the indentation is correct in my config file.

[jsoriano (Jaime Soriano)]

@kellen.hopwood I saw you edited the sample data, take into account that with your configuration the pattern XXXXX must appear as the first line of your multiline event.

Another option you could try to avoid needing to add these placeholders is to use a pattern that matches with the usual requests you have, i.e. something like this:

  multiline:
     pattern: '^(GET|HTTP/)' 
     negate: true
     match: after

[kellen.hopwood]

Hi @jsoriano, I have been digging through my logs, and even when attempting to match the first line, with HTTP or POST or HEAD, GET, ect, I still get the entire file being sent pretty much all in one big log, and I was hoping that I could separate each packet. However with Negate True I can't seem to get the individual packets what are being separated by the matching regex. With negate off, I get each line of the log sent individually. It seems like it maybe a regex parse issue? I think I'm going to try and insert an EOF at the end of every file and the use the flush command on that.

[kellen.hopwood]

I tried to insert a flush, point at the end of each log, however it still isn't recognizing either point, it's just grabbing all the data because it registers it all as a false, and because negate is on, it inverts that and registers it all as true. Any advice @jsoriano?