Multiline in filebeat

[Manasa (Manasa Devadas)]

Multiline is clubbing unmatched lines to another event. Please let me know how I can exclude those lines. I want only the XML in the output.

Example log file

line1
line2
<Errors>
inside the xml1
   inside the xml1-abc
</Errors>
outsideXML1
outsideXML1-line2
<Errors>
inside the xml2
   inside the xml2-abc
</Errors>
outside the xml2

pattern in filebeat.yml

  multiline:
    pattern: '<Errors>'
    negate: true
    match: after
    flush_pattern: '</Errors>'

Output file

{"@timestamp":"2017-11-16T06:09:41.821Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"message":"line1\nline2","source":"/opt/servers/logs/test1.log","offset":12,"prospector":{"type":"log"},"beat":{"name":"ip-10-120-212-122.ap-south-1.compute.internal","hostname":"ip-10-120-212-122.ap-south-1.compute.internal","version":"6.0.0"}}
{"@timestamp":"2017-11-16T06:09:41.821Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/opt/servers/logs/test1.log","offset":70,"message":"\u003cErrors\u003e\ninside the xml1\n   inside the xml1-abc\n\u003c/Errors\u003e","prospector":{"type":"log"},"beat":{"name":"ip-10-120-212-122.ap-south-1.compute.internal","hostname":"ip-10-120-212-122.ap-south-1.compute.internal","version":"6.0.0"}}
{"@timestamp":"2017-11-16T06:09:41.821Z","@metadata":**{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/opt/servers/logs/test1.log","offset":100,"message":"outsideXML1\noutsideXML1-line2","prospector":{"type":"log"},"beat":{"name":"ip-10-120-212-122.ap-south-1.compute.internal","hostname":"ip-10-120-212-122.ap-south-1.compute.internal","version":"6.0.0"}}**
{"@timestamp":"2017-11-16T06:09:41.822Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"beat":{"name":"ip-10-120-212-122.ap-south-1.compute.internal","hostname":"ip-10-120-212-122.ap-south-1.compute.internal","version":"6.0.0"},"source":"/opt/servers/logs/test1.log","offset":158,"message":"\u003cErrors\u003e\ninside the xml2\n   inside the xml2-abc\n\u003c/Errors\u003e","prospector":{"type":"log"}}
{"@timestamp":"2017-11-16T06:09:41.822Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/opt/servers/logs/test1.log","offset":176,"message":"outside the xml2\n","prospector":{"type":"log"},"beat":{"name":"ip-10-120-212-122.ap-south-1.compute.internal","hostname":"ip-10-120-212-122.ap-south-1.compute.internal","version":"6.0.0"}}

I want only second and fourth lines in the output.

[steffens (Steffen Siering)]

For this use-case you would need to have a start + flush pattern, such that multiline is only active after the start-pattern has been hit. The multiline as is, is always active. Feel free to file an enhancement request.

[Manasa (Manasa Devadas)]

Thanks for the response. I resolved it by adding include_lines = <Errors>
I will try start + flush as well.