Filebeat Multiline publishing to Kibana as singlelines

[phillipgibb (phillipgibb@gmail.com)]

Hi,

I am using FileBeat 5.1.1

I have a portion of a log file:

2 2014-01-01 00:08:27.517 XML.Handler[3]: Received:
<request version="00">
	<date>20140101</date>
	<time>000827</time>
</request>
2 2014-01-01 00:08:27.564 ConfirmCustomerRequestHandler[0]: POST /services HTTP/1.0

My multiline config is:

multiline:
  pattern: '^[0-9]{1}\s[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}\s'
  match: after
  negate: true
  timeout: 15s
  max_lines: 1000

In Kibana I see the 6 lines in different entries with the same timestamp.

I tested this in the Go Playground and it works:

matches	line
false	2 2014-01-01 00:08:27.517 XML.Handler[3]: Received:
true	<request version="00">
true		<date>20140101</date>
true		<time>000827</time>
true	</request>
false	2 2014-01-01 00:08:27.564 ConfirmCustomerRequestHandler[0]: POST /services HTTP/1.0

Is there something I am missing? If I include the lines matching the pattern then all the multilines get dropped.
thanks
Phillip

[andrewkroh]

Double check the indentation in your config file. I tested with:

filebeat.prospectors:
- paths: [input.txt]
  multiline:
    pattern: '^[0-9]{1}\s[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}\s'
    match: after
    negate: true
    timeout: 15s
    max_lines: 1000

output.console.pretty: true
logging.level: debug

where input.txt contained the snippit you gave.

The output was:

{
  "@timestamp": "2017-01-04T16:02:47.395Z",
  "beat": {
    "hostname": "macbook",
    "name": "macbook",
  },
  "input_type": "log",
  "message": "2 2014-01-01 00:08:27.517 XML.Handler[3]: Received:\n\u003crequest version=\"00\"\u003e\n    \u003cdate\u003e20140101\u003c/date\u003e\n    \u003ctime\u003e000827\u003c/time\u003e\n\u003c/request\u003e",
  "offset": 136,
  "source": "input.txt",
  "type": "log"
}
{
  "@timestamp": "2017-01-04T16:02:47.395Z",
  "beat": {
    "hostname": "macbook",
    "name": "macbook",
  },
  "input_type": "log",
  "message": "2 2014-01-01 00:08:27.564 ConfirmCustomerRequestHandler[0]: POST /services HTTP/1.0",
  "offset": 220,
  "source": "input.txt",
  "type": "log"
}

[phillipgibb (phillipgibb@gmail.com)]

@andrewkroh

output.console.pretty: true

it was the indentation. as much as I had stared at the indentation I could not convince myself that was the issue,
It really was because of all the comments in the yml; you config, without the comments, made it clear to me.

Thanks