Multiline pattern works on The Go Playground but not filebeat

Elastic Stack Beats
1

I am trying to parse xml logs which are in multiline format. The pattern works fine when I checked on The Go Playground , but not working when I actually start filebeat. Filebeat version (5.3.1) running on RHEL7

Input log:

<05/12/2017 08:25:19.218 EDT> [DEBUG] - [Ver: 1.0.0-SNAPSHOT ] `....some message...`<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<service>
     <..>.....</..>
     <..>.....</..>
</service>

Pattern:

  multiline.pattern: '^(\<[\d]{2})'
  multiline.negate: true
  multiline.match: after

Go Playground:

matches	line
false	<05/12/2017 08:25:19.218 EDT> [DEBUG] - [Ver: 1.0.0-SNAPSHOT ] ..............<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
true	<service>
true	      <..>
...

Filebeat logs in debug mode:

    2017-05-12T13:00:48-04:00 DBG  Publish: {
      ............
      "input_type": "log",
      "message": "\u003c05/12/2017 08:25:19.218 EDT\u003e [DEBUG] - [Ver: 1.0.0-SNAPSHOT ] ...... \u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?\u003e",
      "offset": 280,
      "source": "...........",
      "type": "xyz"
    }
   2017-05-12T13:00:48-04:00 DBG  Publish: {
      ............
      "input_type": "log",
      "message": "\u003cservice\u003e",
      "offset": 280,
      "source": "...........",
      "type": "xyz"
    }

As the log shows, the second log line is being taken as separate event, instead of appending to first line. this is happening for all the lines and so each line is being read as separate event.

Any solution?

2

can you share the more complete config? If indentation is off or multiline not put into the prospector it's not enabled.

Have you tried with filebeat-multiline-tester? If it work for filebeat-multiline-test, but not filebeat, can you try to upgrade to 5.4 or downgrade to 5.2?

This pattern '^\<\d{2} should also work + should use a better performaning custom matcher in 5.3.

3

Thanks for the reply @steffens. It was actually my mistake, I didn't apply this configuration to required environment.
Its working perfectly.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.