How to parse irregular xml logs

Elastic Stack Logstash
1

Hi

How to parse irregular xml logs

sample log is shown below.

DEBUG 2018/08/16 13:01:17.307842 Response XML:- <Header version="1.0"msg_type="SERVICE_REQUEST">GR_IFA_IN_B2C_127_0_0_1_01<Operator_details operator_w_name="svt" name="SVN"></Operator_details><Control_AccountInfo></Control_AccountInfo><PG_AccountInfo></PG_AccountInfo><Service_Request name="get_limit"></Service_Request><Duration_Info Initial_time="1534404677" Subsystem_name="IFA_IN"></Duration_Info><Duration_Info Final_time="1534404677307" Initial_time="1534404677154" Total_time="153" Subsystem_name="SM"></Duration_Info><Duration_Info Final_time="1534404677301" Initial_time="1534404677159" Total_time="142" Subsystem_name="TP"></Duration_Info><Duration_Info Final_time="1534404677200" Initial_time="1534404677164" Total_time="36" Subsystem_name="USR"></Duration_Info><Duration_Info Final_time="1534404677293" Initial_time="1534404677211" Total_time="82" Subsystem_name="WL"></Duration_Info>

2

What do you mean by irregular, i.e. what exactly are you having problems with?

4

Logs with different values or fields (problem iam facing is to define Xpath in logstash xml filter because there is no fixed values or fields).

TRACE 2018/08/16 12:44:33.496034 Response XML is <zero_System_Interface version="1.0"><SystemService_Response version="1.0" name="GetTxnInfo"><Response_Info>0030No record found.01DECLINED000000000696Transaction Record not found</Response_Info></SystemService_Response></zero_System_Interface></receiver
TRACE 2018/08/16 12:45:33.496084 Response XML is <zero_version="1.0"><zero_System_Interface version="1.0"><SystemService_Response version="1.0" name="GetTxnInfo">0030No record found.01DECLINED000000000696Transaction Record not found</SystemService_Response></zero_System_Interface></zero_version>

5

Well, if the XML has a varying schema you'll have problems defining XPath expressions. Perhaps you should just parse the whole XML into the event?

6

While parsing whole XML into event, fields are missing

7

What XML are you parsing? What result do you get?

8

Iam using this filter....

filter {
xml {
source => "message"
target => "parser"
}
}

My logs are

@timestamp August 20th 2018, 09:51:42.715
t @version 1
t _id JpmRVWUBQofrep64Wv4us
t _index filebeat-2018.08.20

_score -

t _type doc
t beat.hostname grcentos
t beat.name grcentos
t beat.version 6.3.2
t host.name grcentos
t input.type log
t message TRACE 2018/08/16 12:44:33.496034 Response XML is <receiver_INTERFACE version="1.0"><receiver_System_Interface version="1.0"><SystemService_Response version="1.0" name="GetTxnInfo"><Response_Info>0030No record found.01DECLINED000000000696Transaction Record not found</Response_Info></SystemService_Response></receiver_System_Interface></receiver_INTERFACE>

offset 1,575

? parser.receiver_System_Interface {
"SystemService_Response": [
{
"version": "1.0",
"name": "GetTxnInfo",
"Response_Info": [
{
"HostDescription": [
"No record found."
],
"HostCode": [
"0030"
],
"ClassCode": [
"01"
],
"SystemDescription": [
"Transaction Record not found"
],
"SystemCode": [
"000000000696"
],
"ClassDescription": [
"DECLINED"
]
}
]
}
],
"version": "1.0"
}
? parser.version 1.0
t prospector.type log
t source /var/log/test.log
t tags beats_input_codec_plain_applied

9

Hello Magnus please reply on above query

10

Okay, so it appears to parse the XML without any problems. What's the desired outcome?

11

There is no fields without xpath...

I need fields like as shown below from above parsed xml logs witout Xpath

ClassCode: 01
SystemDescription:Transaction Record not found
SystemCode:000000000696

12

The fields are there, they're just not rendered in the same way in Kibana. You can move fields with a mutate filter, but since the XML schema isn't fixed but somehow dynamic you'll have to use a ruby filter. In what way are the XML documents different?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.