Logstash expecting \n for it to process an XML

Elastic Stack Logstash
1

Hello Team,

I am trying to parse through an XML file and read only a single tag out of the entire XML using the Grok Patterns.

My grok pattern looks like this. Its able to parse through an XML when its properly indented, as there's a new line after each closing tag. But when the file comes with no spaces in between consecutive tags, this pattern does not work. Could anyone help here?

filter {
#ignore log comments
if [message] =~ "^#" {
drop {}
}

grok {
patterns_dir => "./patterns"
match => ["message", "%{DATA:extras}<LoadID%{DATA:extra}>%{DATA:ASNNumber}%{GREEDYDATA:behind}"]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
locale => "en"
}
}

Second filter

filter {
if "_grokparsefailure" in [tags] {
drop { }
} else {
# on success remove the message field to save space
mutate {
remove_field => ["message", "timestamp", "extra", "extras", "behind"]
}
}
}

2

I suggest you use an xml filter to parse XML.

3

I am not really sure how to go about about it?
Can you please suggest changes in my existing filters?
I only need to read the value of the tag
<LoadID></LoadID> or <LoadID xmlns=""></LoadID>

4

I can't help without knowing what the data looks like. What does a stdout { codec => rubydebug } output produce?

5

This works fine for me.

<tns:ASNAck xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0">
    <MessageHeader>
        <MessageID>20170704080189</MessageID>
        <MessageTimeStamp>2017-07-04T20:17:30Z</MessageTimeStamp>
        <SenderID>KNN_DAO_MXC</SenderID>
        <ReceiverID>GOLF_DAO</ReceiverID>
        <MessageType>InboundASNAck</MessageType>
        <CorrelationID>2d383736363033383337333530313338</CorrelationID>
    </MessageHeader>
    <MasterWaybill>C211</MasterWaybill>
    <LoadID>10112275912A02</LoadID>
    <Accept>true</Accept>
    <ReasonCode>0</ReasonCode>
    <ReasonDescription/>
</tns:ASNAck>

This doesn't work:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?><tns:ASNAck xmlns:tns='http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'><MessageHeader><MessageID>3031999515</MessageID><MessageTimeStamp>2017-09-12T06:37:36Z</MessageTimeStamp><SenderID>XBHSNS</SenderID><ReceiverID>GOLF_DAO</ReceiverID><MessageType>ACKACKACK</MessageType><CorrelationID>2d323537383935353034383933383135</CorrelationID></MessageHeader><Masterbill>G829441</Masterbill><LoadID>Jitesh555</LoadID><Accept>true</Accept><ReasonCode/><ReasonDescription/></tns:ASNAck>

6

That's not the output from stdout { codec => rubydebug }. For best results please answer my questions. I'd also like to see the complete configuration (specifically your inputs).

7 
input {
  beats {
    port => 5045
    type => 'iis'
  }

}

This is my input. There is nothing in the ruby debug as the pattern doesn't match

8

There is nothing in the ruby debug as the pattern doesn't match

Then stop dropping non-matching events so that all events are routed to the stdout output

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.