Filebeat Process multilne XML

[Sourabh_Narang]

I have a log file where I am printing request and response XML bodies. This log file also has some additional lines that are not XML. Example below --

 ===========================================
<request>
  <data>
     abc
</data>
</request>
some random lines concerning the processing of request..
fha;js;fkgak;gj;kj;a
jgahkg;gaj;
Now response follows 
<response>
    <output>
    def
 </output>
</response>
===================================================

How to use use filebeat multiline feature to parse such a data and have each XML as a single event to be sent to logstash?

[helloworld (Abhinaythurlapati)]

I also faced this problem once. I couldn't find the solution. Modifying the log file at the source is one such option. But not always recommended.

[andrewkroh]

I don't think this can be done with Filebeat alone.

One approach would be to use multiline in Filebeat to group all the lines from the request and response into one event.

filebeat.prospectors:
  - paths: ['/my/file.log']
    multiline.pattern: ^<request>
    multiline.negate: true
    multiline.match: after

Then in either Logstash or Ingest Node, use grok to parse the request, response, and text about the request into separate fields.

And after grok'ing out the request and response you could apply an xml filter to parse those messages.