XML filter plugin is not working!

[Ritesh_Sharma]

Hi People ,

I am pulling some stream of XML files from Kafka topic and trying to fetch one of the element of XML file through Logstash using XML filter plugin but unable to do so ; mentioning details below ;

XML block of stream :

<?xml v="1.0" enc="UF-8"?>

<\ns0:LogRequest xmlns:ns0="http://SOME/URL/SOME.xsd">
<\ns1:Header xmlns:ns1="http://SOME/URL/SOME1.xsd">
<\ns1:ApplicationID>ABC</ns1:ApplicationID>
<\ns1:ComponentName>XYZ</ns1:ComponentName>
<\ns1:Hostname>MYHOST</ns1:Hostname>
<\ns1:Timestamp>2017-02-27T23:07:44.318+05:30</ns1:Timestamp>
<\ns1:TransactionDomain>Some Domain </ns1:TransactionDomain>
<\ns1:TransactionType>Some Type</ns1:TransactionType>
<\ns1:Message>Some Message </ns1:Message>
<\ns1:AltKey>\n
<\ns1:AltKeyName>LOG</ns1:Alt\KeyName>
<\ns1:AltKeyValue>Some Key </ns1:AltKeyValue>\n
</ns1:AltKey>\n
</ns1:Header>\n
<\TimeDuration>1488217064317</TimeDuration>\n
<\Status>Some Status </Status>\n
<\Audit>true\n</ns0:LogRequest> "

I want to fetch the value of "Timestamp" and save it as new field but unable to do so ,

Filter block of Logstash.conf :

filter {
xml {
store_xml => "false"
remove_namespaces => "true"
source => "message"
xpath => [ "/LogRequest/Timestamp/text()", "mytime" ]
}

mutate {
add_field => ["Timestamp", "%{mytime}"]
}
}

In output, new field "Timestamp" is getting added but value of same remain as %{mytime} i.e. static string .

KIndly help !!

P.S. : I am new to Xpath .
I have appended the xml with "" at every line, as It was difficult to post normal xml here.

[magnusbaeck (Magnus Bäck)]

I think it's the namespace that's preventing this from working for you. I think the xml filter has an option for dealing with this. It was discussed here a few weeks ago.

I have appended the xml with "" at every line, as It was difficult to post normal xml here.

Select the text and click the </> button on the toolbar to format it as preformatted text that keeps the angle brackets.

[Ritesh_Sharma]

Hi Magnus ,

are you talking about "namespaces" configuration ; i tried with following conf but the problem still persist ,

filter {

xml {
store_xml => "false"
remove_namespaces => "false"
source => "message"
namespaces => {
"ns0" => "http://SOME/URL/SOME.xsd"
"ns1" => "http://SOME/URL/SOME1.xsd"
}
xpath => [ "/LogRequest/Timestamp/text()", "mytime" ]
}

mutate {
add_field => ["time_stamp", "%{mytime}"]
}

}

I also tried with remove_namespaces => "true" with same conf , but still getting the "time_stamp" => "%{mytime}" as output .

Could you kindly provide any link to the last similar question , where you have mentioned the solution for such issue .

[Ritesh_Sharma]

Well, i resolved this issue by doing following steps ,

• Removed the namespaces from Logstash processing ;

• Updated the X-path for Timestamp element ( i missed the header element , earlier )

;
so now my configuration look like ,

filter {

xml {
store_xml => "false"
remove_namespaces => "true"
source => "message"
xpath =>["//[localname()='LogRequest']/*/[localname()='Timestamp']/text()", "mytime" ]

 }

}

So , essentially the updated X-path was the solution.

Cheers,