Parse XML file question

terrymu · March 26, 2018, 2:07am

Hi experts,

I have parsed a XML file by using Xpath, but some field can't be detect, would you please help me check it? Thank you.

The input file:

<?xml version="1.0" encoding="UTF-8"?>
<OMeS version="2.3">
  <PMSetup startTime="2018-03-09T08:15:00.000+01:00" interval="5">
    <PMMOResult>
      <MO dimension="network_element">
        <DN>NTAS-tas01/HOSTNAME-cbam-4c63de7aeae2460289cd4197dc7-admintd-node-0/DBTYPE-cmdb</DN>
      </MO>
      <PMTarget measurementType="DBMEAS">
        <M704B1C1>32</M704B1C1>
        <M704B1C2>188</M704B1C2>
        <M704B1C3>0</M704B1C3>
        <M704B1C4>0</M704B1C4>
      </PMTarget>
    </PMMOResult>
    <PMMOResult>
      <MO dimension="network_element">
        <DN>NTAS-tas01/HOSTNAME-cbam-4c63de7aeae2460289cd4197dc7-admintd-node-1/DBTYPE-cmdb</DN>
      </MO>
      <PMTarget measurementType="DBMEAS">
        <M704B1C1>30</M704B1C1>
        <M704B1C2>116</M704B1C2>
        <M704B1C3>0</M704B1C3>
        <M704B1C4>1</M704B1C4>
      </PMTarget>
    </PMMOResult>

=============================================================

The parsed pattern

xml {
                        source => "message"
                        target => "parsed"

                        xpath => [
                                "/OMeS/PMSetup/@startTime", "audit_time",
                                "/OMeS/PMSetup/PMMOResult/MO/@dimension", "ELKNE",
                                "/OMeS/PMSetup/PMMOResult/MO/DN/text()", "ELKDN",
                                "/OMeS/PMSetup/PMMOResult/PMTarget/@measurementType", "ELKPMtype",
                                "/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo"
                       ]

=======================================================================

Now the major question is:
"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo" can't get this <M704B1C1>32</M704B1C1> value!

wwalker · March 26, 2018, 2:21am

What's the output look like? Is there an error message in either Logstash or Elasticsearch? Have you looked at the debug level logs?

terrymu · March 26, 2018, 2:33am

Thanks a lot to answer my question.
How can I check debug level logs?
I only have the output from kibana, it shows as below:

wwalker · March 26, 2018, 2:41am

To turn on debug logging in Logstash, put log.level: debug in logstash.yml. For Elasticsearch, logger.level: debug in elasticsearch.yml. These can be very chatty logs so be sure to comment the settings out after you finish troubleshooting.

So two things to start: open an event that makes it into Kibana and click the arrow to expand the event. Then click on JSON and post a capture of it on here. Second, post your full pipeline on here so we can better understand the whole process the data is going through.

terrymu · March 26, 2018, 3:22am

Hi wwalker,

The Json data is too large to upload. from the Json, there is no area populated from OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()

The whole pattern:
input {

        file {
                path => ["/home/admin/log/NTASlog/*xml"]
                start_position => "beginning"
                type => "pmxmllog"
                sincedb_path => "/home/admin/log/NTASlog/.sincedb_file"
                codec => multiline {
                        pattern => "<?xml version"
                        #auto_flush_interval => 5
                        #max_lines => 60000
                        what => "previous"
                        negate=> true
                }

        }

}

filter {

        if [type] == "pmxmllog" {

                xml {
                        source => "message"
                        target => "parsed"

                        xpath => [
                                "/OMeS/PMSetup/@startTime", "audit_time",
                                "/OMeS/PMSetup/PMMOResult/MO/@dimension", "ELKNE",
                                "/OMeS/PMSetup/PMMOResult/MO/DN/text()", "ELKDN",
                                "/OMeS/PMSetup/PMMOResult/PMTarget/@measurementType", "ELKPMtype",
                                "/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget[@measurementType]/M704B1C1/text()", "ELKPVoo",
                                #"/OMeS/PMSetup/PMMOResult/PMTatget//M704B1C1/text()", "ELKPVooo"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget[1]/text()", "PV2"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C3[0]", "PV3",
                                #"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C4[1]", "PV4"
                        ]


                }
        }

        #mutate {
        #       split => [ "NE", "," ]
        #}

        #mutate {
        #        split => [ "DN", "," ]
        #}


        date {
                match => [ "audit_time", "YYYY-MM-DD HH:MM:SS.SSS" ]
#               target => "audit_time"
#               locale => "en_US"
                timezone =>  "UTC"

        }

}


output {
        elasticsearch {
                        hosts => ["localhost:9200"]
        }
}

And some output log from ela

terrymu · March 26, 2018, 5:55am

Hi wwalker,

I consider I have found the issue....
"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()"
The "target" I have type to "tatget"...

Thanks to your request of "post your full pipeline on here", I have found the issue.
Thanks a lot for your supports.

wwalker · March 27, 2018, 2:33am

doh! Two sets of eyes took a long time to spot that one.

terrymu · March 27, 2018, 3:22am

Hi wwalker,

I have met another issue, can you provide some helps for me?

Thank you.

terrymu · March 27, 2018, 3:23am

Hi wwalker,

I have met another issue, can you provide some helps for me?

Thank you.

system · April 24, 2018, 3:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.