Parse XML file question

Elastic Stack Logstash
1

Hi experts,

I have parsed a XML file by using Xpath, but some field can't be detect, would you please help me check it? Thank you.

The input file:

<?xml version="1.0" encoding="UTF-8"?>
<OMeS version="2.3">
  <PMSetup startTime="2018-03-09T08:15:00.000+01:00" interval="5">
    <PMMOResult>
      <MO dimension="network_element">
        <DN>NTAS-tas01/HOSTNAME-cbam-4c63de7aeae2460289cd4197dc7-admintd-node-0/DBTYPE-cmdb</DN>
      </MO>
      <PMTarget measurementType="DBMEAS">
        <M704B1C1>32</M704B1C1>
        <M704B1C2>188</M704B1C2>
        <M704B1C3>0</M704B1C3>
        <M704B1C4>0</M704B1C4>
      </PMTarget>
    </PMMOResult>
    <PMMOResult>
      <MO dimension="network_element">
        <DN>NTAS-tas01/HOSTNAME-cbam-4c63de7aeae2460289cd4197dc7-admintd-node-1/DBTYPE-cmdb</DN>
      </MO>
      <PMTarget measurementType="DBMEAS">
        <M704B1C1>30</M704B1C1>
        <M704B1C2>116</M704B1C2>
        <M704B1C3>0</M704B1C3>
        <M704B1C4>1</M704B1C4>
      </PMTarget>
    </PMMOResult>

=============================================================

The parsed pattern

xml {
                        source => "message"
                        target => "parsed"

                        xpath => [
                                "/OMeS/PMSetup/@startTime", "audit_time",
                                "/OMeS/PMSetup/PMMOResult/MO/@dimension", "ELKNE",
                                "/OMeS/PMSetup/PMMOResult/MO/DN/text()", "ELKDN",
                                "/OMeS/PMSetup/PMMOResult/PMTarget/@measurementType", "ELKPMtype",
                                "/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo"
                       ]

=======================================================================

Now the major question is:
"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo" can't get this <M704B1C1>32</M704B1C1> value!

2

What's the output look like? Is there an error message in either Logstash or Elasticsearch? Have you looked at the debug level logs?

3

Thanks a lot to answer my question.
How can I check debug level logs?
I only have the output from kibana, it shows as below:

elk1
elk1.png1165×587 56.3 KB

4

To turn on debug logging in Logstash, put log.level: debug in logstash.yml. For Elasticsearch, logger.level: debug in elasticsearch.yml. These can be very chatty logs so be sure to comment the settings out after you finish troubleshooting.

So two things to start: open an event that makes it into Kibana and click the arrow to expand the event. Then click on JSON and post a capture of it on here. Second, post your full pipeline on here so we can better understand the whole process the data is going through.

5

Hi wwalker,

The Json data is too large to upload. from the Json, there is no area populated from OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()

The whole pattern:
input {

        file {
                path => ["/home/admin/log/NTASlog/*xml"]
                start_position => "beginning"
                type => "pmxmllog"
                sincedb_path => "/home/admin/log/NTASlog/.sincedb_file"
                codec => multiline {
                        pattern => "<?xml version"
                        #auto_flush_interval => 5
                        #max_lines => 60000
                        what => "previous"
                        negate=> true
                }

        }

}

filter {

        if [type] == "pmxmllog" {

                xml {
                        source => "message"
                        target => "parsed"

                        xpath => [
                                "/OMeS/PMSetup/@startTime", "audit_time",
                                "/OMeS/PMSetup/PMMOResult/MO/@dimension", "ELKNE",
                                "/OMeS/PMSetup/PMMOResult/MO/DN/text()", "ELKDN",
                                "/OMeS/PMSetup/PMMOResult/PMTarget/@measurementType", "ELKPMtype",
                                "/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()", "ELKPVo"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget[@measurementType]/M704B1C1/text()", "ELKPVoo",
                                #"/OMeS/PMSetup/PMMOResult/PMTatget//M704B1C1/text()", "ELKPVooo"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget[1]/text()", "PV2"
                                #"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C3[0]", "PV3",
                                #"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C4[1]", "PV4"
                        ]


                }
        }

        #mutate {
        #       split => [ "NE", "," ]
        #}

        #mutate {
        #        split => [ "DN", "," ]
        #}


        date {
                match => [ "audit_time", "YYYY-MM-DD HH:MM:SS.SSS" ]
#               target => "audit_time"
#               locale => "en_US"
                timezone =>  "UTC"

        }

}


output {
        elasticsearch {
                        hosts => ["localhost:9200"]
        }
}

And some output log from ela

elk2
elk2.png1180×87 6.6 KB

elk3

6

Hi wwalker,

I consider I have found the issue....
"/OMeS/PMSetup/PMMOResult/PMTatget/M704B1C1/text()"
The "target" I have type to "tatget"...

Thanks to your request of "post your full pipeline on here", I have found the issue.
Thanks a lot for your supports.

7

doh! Two sets of eyes took a long time to spot that one.

8

Hi wwalker,

I have met another issue, can you provide some helps for me?

Thank you.

9

Hi wwalker,

I have met another issue, can you provide some helps for me?

Thank you.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.