Filebeat multiline patterns not working

Elastic Stack Beats
1

filebeat v1.2.2 (64 bit)
OS: centos 6.7

multiline: --> patterns: --> '-' --> pattern: (not working)
multiline: --> pattern: (working)

Found detail here:
(Enhanced LS Config)
https://github.com/elastic/filebeat/issues/301

Is this supposed to work or feature not available yet.

2

Multline support is part of the 1.2 release. The docs can be found here: https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html#multiline

I'm not sure what you mena with your patterns / pattern above?

3

please add some sample logs you want to merge + regex you've configured. I can't even tell what you're trying to do.

6

Pasted image1326×724 24.6 KB

7

You've got text instead of pictures. Here is a script to test your regexes, but I'm not too keen to type out all these pictures.

Put text in between 3 backticks '`' or use the </> button on inserted text in order to code-format your text.

8

I just want to know that the feature is available in Filebeat? The feature I mean more then one pattern in multiline.

9

Thanks Steffen for your trick for formating.

multiline:
  pattern: '^Encore.DMS.Svc-'
  negate: true
  match: after

This above config works for me for single condition, but not sure about more then one condition like I wrote below.

multiline:
  patterns:
    -
      pattern: '^Encore.DMS.Svc-'
      negate: true
      match: after
    -
      pattern: '^Encore.DMS.Svc-\.*RESPONSE:'
      negate: false
      match: after
10

multiple conditions are not supported yet. But one can use the | (OR-)regex operator in in pattern.

11

Log content in proper format:

Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:246|AgentFirstName : 
Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:315|REQUEST:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Body ...>
    <h2h-dms-request xmlns=...>
	...
    </h2h-dms-request>
  </s:Body>
</s:Envelope>
Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:730|RESPONSE:
<soapenv:Envelope ...>
  <s:Header xmlns:s=... />
  <soapenv:Body>
    <xrsi:h2h-dms-reply xmlns:xrsi=...>
	...
    </xrsi:h2h-dms-reply>
  </soapenv:Body>
</soapenv:Envelope>
Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:730|Return value:
Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:731|	
Encore.DMS.Svc-6.2-0-1000013856-INFO|08:08:30:731|Code : 000
12

Thanks for your help.
I already have this config and it is working as expected.

multiline:
  pattern: '(^Encore.DMS.Svc-)|(^Encore.DMS.Web-)|(^Encore.DASMonitor-)'
  negate: true
  match: after

But for my last condition I need to negate inside the regex. Which I think is not allowed or will not work because negate handled in separate line like "negate: true".
"! (^Encore.DMS.Svc-\.*RESPONSE:)"
Something like above can be doable or any other way this can be achieved in Filebeat?

13

Do I understand you correctly? You just want to merge REQUEST plus RESPONSE attributes?

Check out this solution: https://play.golang.org/p/IS8wDp1h6F
Regex is ^([[:space:]]|<|(.*\|){2}RESPONSE)
First sub-term matches a string starting with whitespace, the second sub-term matches a line starting with < and the third subfilter matches ... | ... |RESPONSE without even checking the content of the columns.