How to write a filter for a file containing a mix of xml and non xml messages

Elastic Stack Logstash
1

Hello,
I have a log file which is a mix of xml and regular (non xml) lines. I need to apply grok filer + xml filter to the lines that has xml block and apply only grok filter to the regular lines. For that I need help in extracting the xml block using the logstash filter but I am not able to do that. If I use the xml filter like:
xml { source => "message" target => "xml" }
it gets applied to all the messages and the lines without xml block in it are either dropped or comes back as _xmlparsefailure, also the lines with xml block are parsed correctly.
I need help in figuring out how to extract the xml block and apply xml filter only if it exists in message.

Thank you.

2

You need to determine whether the [message] field contains XML. The easiest way to do that is with an xml filter. If the event gets tagged with "_xmlparsefailure" then it is not valid XML, so you can make additional processing conditional upon that.

3

Ahh, got it. So instead of finding a xml block and then applying the xml filter, you are suggesting finding the regular messages using the "_xmlparsefailure" tag and then do what I need to do.

Thank you.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.