# Parse inconsistent XML trace using logstash

**URL:** <https://discuss.elastic.co/t/parse-inconsistent-xml-trace-using-logstash/236985>\
**Category:** Logstash\
**Created:** [June 13, 2020, 3:44pm UTC](https://discuss.elastic.co/t/parse-inconsistent-xml-trace-using-logstash/236985 "2020-06-13T15:44:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![joshua.stanley](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@joshua.stanley](https://discuss.elastic.co/u/joshua.stanley)\
**Post date:** [June 13, 2020, 3:44pm UTC](https://discuss.elastic.co/t/parse-inconsistent-xml-trace-using-logstash/236985/1 "2020-06-13T15:44:59Z")

</div>

I am very new to ELK , need your guidance in my below usecase development.

We have lot of XML files generated from source server(around 55GB per day )..but, from each file, we need only very few lines.

There are 2 parameters(CO\_ID\_PUB,DIRNUM) are of interest in each of the XML trace containing transactions.

But, each of the transaction may or may not contain both of them. For example:

1. Both CO\_ID\_PUB,DIRNUM are available

```auto
<COMMAND name="AAA.READ" timestamp="1574339699569" so="103">
<SVLOBJECT>
            <STRING name="CO_ID_PUB" val="FFFFFFF"/>
        </SVLOBJECT>
        <RESULT>
          ...
          <STRING name="DIRNUM" val="FFFFFFF"/>
          ....
</COMMAND>
<TX_COMMIT timestamp="1574339699585" so="103"/>

```

1. Only CO\_ID\_PUB available

```auto
<COMMAND name="BBB.READ" timestamp="1574339699569" so="103">
<SVLOBJECT>
            <STRING name="CO_ID_PUB" val="FFFFFFF"/>
        </SVLOBJECT>
        <RESULT>
          ...
          ....
</COMMAND>
<TX_COMMIT timestamp="1574339699585" so="103"/>

```

1. Only DIRNUM available

```auto
<COMMAND name="CCC.READ" timestamp="1574339699569" so="103">
<SVLOBJECT>
            <STRING name="DIRNUM" val="FFFFFFF"/>
        </SVLOBJECT>
        <RESULT>
          ...
          ....
</COMMAND>
<TX_COMMIT timestamp="1574339699585" so="103"/>

```

1. Neither CO\_ID\_PUB,DIRNUM are not available

Each transaction starting with this below pattern:

```auto
<COMMAND name="AAA.READ" timestamp="1574339699569" so="103">
<SVLOBJECT>
            <STRING name="CO_ID_PUB" val="FFFFFFF"/>
        </SVLOBJECT>
        <RESULT>
....
</COMMAND>
<TX_COMMIT timestamp="1574339699585" so="103"/>

```

but as i mentioned in the 4 scenarios, only if the below 2 fields present, then that particular transaction data such as start timestamp & end timestamp to be stored in ES.

Rest of the XML fields need to be dropped.

... ...

Then, i need to store data like below:

{ CO\_ID\_PUB:FFFFFFF, DIRNUM:KKKKK, StartTime:1574339699569 EndTime:1574339699585 name:AAA.READ} or

{ DIRNUM:FFFFFFF, DIRNUM:KKKKK, StartTime:1574339699569 EndTime:1574339699585 name:AAA.READ}

Once this data stored, user will query to get the transaction response time by providing CO\_ID\_PUB or DIRNUM

Can you help me with which logstash filter(xml or grok) and logic to develop parser ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2020, 5:12pm UTC](https://discuss.elastic.co/t/parse-inconsistent-xml-trace-using-logstash/236985/2 "2020-06-14T17:12:09Z")

</div>

Assuming that the \<RESULT\> tags are closed, and that the entire XML is wrapped in a single element (an XML filter will not parse multiple roots) so that your XML items look something like

```
   "message" => "<a> <COMMAND name=\"BBB.READ\" timestamp=\"1574339699569\" so=\"103\"> <SVLOBJECT> <STRING name=\"CO_ID_PUB\" val=\"FFFFFFF\"/> </SVLOBJECT> <RESULT> </RESULT> </COMMAND> <TX_COMMIT timestamp=\"1574339699585\" so=\"103\"/> </a>",

```

then you could parse with an XML filter and use ruby to extract the elements you want. I do not know enough about xpath to say whether it could be done using that.

```
    xml {
        source => "message"
        store_xml => true
        target => "[@metadata][theXML]"
    }
    ruby {
        code => '
            xml = event.get("[@metadata][theXML]")
            if xml
                svl = xml["COMMAND"][0]["SVLOBJECT"][0]
                res = xml["COMMAND"][0]["RESULT"][0]
                if res["STRING"] && res["STRING"][0]["name"] == "DIRNUM"
                    event.set("DIRNUM", xml["COMMAND"][0]["RESULT"][0]["STRING"][0]["val"])
                end
                if svl["STRING"] && svl["STRING"][0]["name"] == "DIRNUM"
                    event.set("DIRNUM", xml["COMMAND"][0]["SVLOBJECT"][0]["STRING"][0]["val"])
                end
                if svl["STRING"] && svl["STRING"][0]["name"] == "CO_ID_PUB"
                    event.set("CO_ID_PUB", xml["COMMAND"][0]["SVLOBJECT"][0]["STRING"][0]["val"])
                end
            end
        '
    }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2020, 5:12pm UTC](https://discuss.elastic.co/t/parse-inconsistent-xml-trace-using-logstash/236985/3 "2020-07-12T17:12:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.