Logstash expecting \n for it to process an XML

(Aggarwal Jitesh94) #1

Hello Team,

I am trying to parse through an XML file and read only a single tag out of the entire XML using the Grok Patterns.

My grok pattern looks like this. Its able to parse through an XML when its properly indented, as there's a new line after each closing tag. But when the file comes with no spaces in between consecutive tags, this pattern does not work. Could anyone help here?

filter {
#ignore log comments
if [message] =~ "^#" {
drop {}

grok {
patterns_dir => "./patterns"
match => ["message", "%{DATA:extras}<LoadID%{DATA:extra}>%{DATA:ASNNumber}%{GREEDYDATA:behind}"]
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
locale => "en"

Second filter

filter {
if "_grokparsefailure" in [tags] {
drop { }
} else {
# on success remove the message field to save space
mutate {
remove_field => ["message", "timestamp", "extra", "extras", "behind"]

(Magnus Bäck) #2

I suggest you use an xml filter to parse XML.

(Aggarwal Jitesh94) #3

I am not really sure how to go about about it?
Can you please suggest changes in my existing filters?
I only need to read the value of the tag
<LoadID></LoadID> or <LoadID xmlns=""></LoadID>

(Magnus Bäck) #4

I can't help without knowing what the data looks like. What does a stdout { codec => rubydebug } output produce?

(Aggarwal Jitesh94) #5

This works fine for me.

<tns:ASNAck xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0">

This doesn't work:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?><tns:ASNAck xmlns:tns='http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'><MessageHeader><MessageID>3031999515</MessageID><MessageTimeStamp>2017-09-12T06:37:36Z</MessageTimeStamp><SenderID>XBHSNS</SenderID><ReceiverID>GOLF_DAO</ReceiverID><MessageType>ACKACKACK</MessageType><CorrelationID>2d323537383935353034383933383135</CorrelationID></MessageHeader><Masterbill>G829441</Masterbill><LoadID>Jitesh555</LoadID><Accept>true</Accept><ReasonCode/><ReasonDescription/></tns:ASNAck>

(Magnus Bäck) #6

That's not the output from stdout { codec => rubydebug }. For best results please answer my questions. I'd also like to see the complete configuration (specifically your inputs).

(Aggarwal Jitesh94) #7
input {
  beats {
    port => 5045
    type => 'iis'


This is my input. There is nothing in the ruby debug as the pattern doesn't match

(Magnus Bäck) #8

There is nothing in the ruby debug as the pattern doesn't match

Then stop dropping non-matching events so that all events are routed to the stdout output

(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.