Filebeat multiline - escaping curly parenthesis

[zoplex]

We have filebeat.yml that processes multiline filter as:

  multiline:
   pattern: '^\{Heap before'
   negate: true
   match: after

we want to start new record whenever the pattern "{Heap before" is encountered;
we see occasionally that records are unnecessary split - almost like '{' was not considered as part of the record start - are we escaping it unnecessary ? Or is that escape syntax incorrect?

Thanks

[zoplex]

.. seems that on some servers first part of the multiline record (about 10 lines) is written to the observed log file - then filebeat takes it and sends it up to logstash although not the full multiline record has been seen; so basically multiline logic applies only as long as the full multiline pattern is already in the log file - filebeat will NOT wait for the rest to be there before it processes it?

[andrewkroh]

@zoplex

are we escaping it unnecessary ? Or is that escape syntax incorrect?

In YAML, single quoted strings do not support (or require) escaping. http://www.yaml.org/spec/1.2/spec.html#id2788097

Then since this string is compiled into a regex it would require escaping if it were a special character or ambiguous. But in either case, escaping or not-escaping it will match. See Go Playground - The Go Programming Language

[zoplex]

Thanks Andrew - so escaping is not an issue here then ... I am assuming that it is the second point above - that filebeat is processing what is written into observer log file - and sending it up even if the full multiline is not there yet - for example if first 10 lines of 15 line multiline are written into the log file now and then the other 5 lines are written 2 minutes later - we will not get complete multiline - filebeat will send first 10 lines and will not wait for two more minutes to get the rest - even if pattern calls for that?

[andrewkroh]

You can try to increase the timeout to test your theory. The default is 5s.

  multiline:
   pattern: '^\{Heap before'
   negate: true
   match: after
   timeout: 60m

The downside is that the longer the timeout, the more latency you have with some log lines showing up in ES.

[zoplex]

Thank you - will do that! Much appreciated!