Hi All, I am using filebeats to try and pull from a text file that contains http logs, the logs are generally not uniform, so I used powershell to create an end of file marker. However when I then try and send the logs into elastic and later kibana, I get one or two massive logs, not the individual components.
filebeat.inputs: - type: log enabled: true paths: - 'C:\Program Files\filebeat\logfile.txt' multiline: pattern: 'XXXXX' negate: true match: after output.logstash: hosts: ["localhost:5044"]
The data looks like this.
GET https://scontent-iad3-1.xx.fbcdn.net/v/t39.2093- 6/36882171_273786243171121_8594480363911249920_n.srt? _nc_cat=0&oh=6dae5ff3122dd34ae1cb4dfccfda7779&oe=5C307E15 HTTP/1.1 Host: scontent-iad3-1.xx.fbcdn.net Connection: keep-alive Origin: https://www.facebook.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36 Accept: */* Referer: https://www.facebook.com/ Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9 AAAAA XXXXX HTTP/1.1 200 OK Last-Modified: Tue, 10 Jul 2018 23:39:25 GMT Content-Type: text/srt Timing-Allow-Origin: * Access-Control-Allow-Origin: * Expires: Tue, 18 Sep 2018 23:54:05 GMT Cache-Control: max-age=1209600, no-transform Date: Sat, 15 Sep 2018 15:14:56 GMT Access-Control-Expose-Headers: X-FB-CEC-Video-Limit Connection: keep-alive Content-Length: 46280 AAAAA XXXXX
My original attempts to have it sort the data have all gone, rather abysmally, I was hoping someone here could give me some quick pointers to get more acclimated. Thanks!