Filebeat Multiline eats up the entire log file


(Maxim Gueivandov) #1

Hello,

I'm trying out the new multiline feature by running FB on an existing log file and having an issue: FB pushes the entire file contents to Logstash as one single message.

Here is my setup:

  • OS: Windows 7 64
  • Filebeat: 1.1.0
  • Logstash: 2.1.1

My test log file (ANSI encoding):

2016-01-19 16:00:00.332 +01:00 [Information][Completed] aaa
2016-01-19 16:00:00.488 +01:00 [Information][Failed] bbb
Something.ValidationException: someting is invalid
   at some method
   at some other method
   something really nasty happened
--- End of stack trace from previous location where exception was thrown ---
   at the end
2016-01-19 16:01:36.520 +01:00 [Information][Completed] ccc

My filebeat.yml:

filebeat:
  prospectors:
    -
      paths:
        - C:/temp/*.log
      input_type: log
      encoding: plain
      fields_under_root: true
      ignore_older: 720h
      document_type: test
      force_close_files: true
      multiline:
          pattern: ^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE} \[
          negate: true
          match: after
...

Filebeat debug output:

2016-02-04T17:30:15+01:00 DBG  Publish: {
  "@timestamp": "2016-02-04T16:30:05.190Z",
  "beat": {
    "hostname": "myhost",
    "name": "myhost"
  },
  "count": 1,
  "input_type": "log",
  "message": "2016-01-19 16:00:00.332 +01:00 [Information] [Provider] [247] [Completed] aaa\r\n2016-01-19 16:00:00.488 +01:00 [Information] [Provider] [176] [Completed] bbb\r\nSomething.ValidationException: someting is invalid\r\n   at some method\r\n   at some other method\r\n   something really nasty happened\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at the end\r\n2016-01-19 16:01:36.520 +01:00 [Information] [Request] [237] [Completed] ccc",
  "offset": 0,
  "source": "C:\\temp\\test.log",
  "type": "test"
}

What am I doing wrong?

Thanks a lot,
MG


(Maxim Gueivandov) #2

Alright, forget this: I wrongly assumed the pattern parameter supports GROK syntax. Which is obviously not the case.
It works well with regexps!


(system) #3