# How to parse and index xml in ES7

**URL:** <https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269>\
**Category:** Logstash\
**Created:** [August 17, 2020, 3:41pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269 "2020-08-17T15:41:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![azer\_gorai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azer_gorai/32/74010_2.png) [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\
**Post date:** [August 17, 2020, 3:41pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/1 "2020-08-17T15:41:20Z")

</div>

I need to index and parse xml in elasticsearsh.  
I need one field to store the xml as raw data and another field that will contain a fvalue retrieved from xpath in the same xml

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 17, 2020, 3:47pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/2 "2020-08-17T15:47:29Z")

</div>

You can use an [xml](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html) filter to parse the XML string. That supports xpath references.

---

<div class="post-metadata">

**Author:** ![azer\_gorai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azer_gorai/32/74010_2.png) [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\
**Post date:** [August 17, 2020, 6:59pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/3 "2020-08-17T18:59:13Z")

</div>

I'm aware that such filter exists. Yet, the documentation is brief i couldn't build a working example from it . Can you show me how i can achieve it and how dummy config file would look like ?  
Thanks 🙂

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 17, 2020, 7:02pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/4 "2020-08-17T19:02:06Z")

</div>

Can you share a sample XML and field and what you are trying to parse?

---

<div class="post-metadata">

**Author:** ![azer\_gorai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azer_gorai/32/74010_2.png) [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\
**Post date:** [August 17, 2020, 7:23pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/5 "2020-08-17T19:23:13Z")

</div>

it looks something like this :

```auto
<dataDocument xmlns="http://www.fpml.org/FpML-5/confirmation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" fpmlVersion="5-9" xsi:schemaLocation="http://www.fpml.org/FpML-5/confirmation ../../fpml-main-5-9.xsd http://www.w3.org/2000/09/xmldsig# ../../xmldsig-core-schema.xsd">
<trade>
<tradeHeader>
<partyTradeIdentifier>
<partyReference href="PartyA"/>
<tradeId tradeIdScheme="http://www.PartyA.com/com-trade-id">ACAVS1234567</tradeId>
</partyTradeIdentifier>
<tradeDate id="TradeDate">2014-04-08</tradeDate>
</tradeHeader>
<commodityPerformanceSwap>
<primaryAssetClass>Commodity</primaryAssetClass>
<productType>Commodity:Energy:Index:Swap:Cash</productType>
<effectiveDate>
<adjustableDate>
<unadjustedDate>2014-04-01</unadjustedDate>
<dateAdjustments>
<businessDayConvention>NONE</businessDayConvention>
</dateAdjustments>
</adjustableDate>
</effectiveDate>
<terminationDate id="TerminationDate">
<adjustableDate>
<unadjustedDate>2014-10-31</unadjustedDate>
<dateAdjustments>
<businessDayConvention>NONE</businessDayConvention>
</dateAdjustments>
</adjustableDate>
</terminationDate>
<commodityVarianceLeg>
<payerPartyReference href="PartyA"/>
<receiverPartyReference href="PartyB"/>
<calculationPeriodsSchedule id="varianceLegCalculationPeriodSchedule">
<periodMultiplier>1</periodMultiplier>
<period>T</period>
<balanceOfFirstPeriod>false</balanceOfFirstPeriod>
</calculationPeriodsSchedule>
<paymentDates>
<adjustableDates>
<unadjustedDate>2014-11-05</unadjustedDate>
<dateAdjustments>
<businessDayConvention>FOLLOWING</businessDayConvention>
<businessCenters>
<businessCenter>USNY</businessCenter>
</businessCenters>
</dateAdjustments>
</adjustableDates>
</paymentDates>
<commodity>
<!-- Note: instrumentId is required only in Confirmation View -->
<instrumentId instrumentIdScheme="http://www.fpml.org/coding-scheme/commodity-reference-price-3-0">MOP-CFR BRAZIL-FMB</instrumentId>
<specifiedPrice>Spot</specifiedPrice>
</commodity>
<notionalAmount id="varianceLegNotionalAmount">
<!-- Note that, because this swap does not have a reinvestment feature on the notional -->
<!-- that the 'amount' below is the notional amount for the Term -->
<currency>USD</currency>
<amount>1000000.00</amount>
<reinvestmentFeature>false</reinvestmentFeature>
</notionalAmount>
<varianceStrikePrice>0.09000</varianceStrikePrice>
<varianceCalculation>
<valuationDates>
<calculationPeriodsScheduleReference href="varianceLegCalculationPeriodSchedule"/>
<dayType>CommodityBusiness</dayType>
<dayDistribution>All</dayDistribution>
</valuationDates>
<annualizationFactor>252</annualizationFactor>
<nAdjustment>false</nAdjustment>
</varianceCalculation>
</commodityVarianceLeg>
<marketDisruption>
<marketDisruptionEvent>AsSpecifiedInMasterAgreement</marketDisruptionEvent>
<disruptionFallbacks>AsSpecifiedInMasterAgreement</disruptionFallbacks>
</marketDisruption>
</commodityPerformanceSwap>
<documentation>
<masterAgreement>
<masterAgreementType>ISDA</masterAgreementType>
<masterAgreementVersion>2002</masterAgreementVersion>
<masterAgreementDate>2010-03-23</masterAgreementDate>
</masterAgreement>
<contractualDefinitions>ISDA2006</contractualDefinitions>
<contractualDefinitions>ISDA2005Commodity</contractualDefinitions>
</documentation>
</trade>
<party id="PartyA">
<partyId partyIdScheme="http://www.fpml.org/coding-scheme/external/iso17442">Party_A_LEI</partyId>
<country>US</country>
<organizationType>SwapDealer</organizationType>
</party>
<party id="PartyB">
<partyId partyIdScheme="http://www.fpml.org/coding-scheme/external/iso17442">Party_B_LEI</partyId>
<organizationType>Non-SD_Non-MSP</organizationType>
</party>
</dataDocument>

```

it follows fpml standard for xml .  
what i'm trying to achieve is for each xml file i need one document in ES that will contain the entire file as a string in one field and its id will be one field that is inside the xml :

perhaps like this:

```auto
{
id: " some field from the xml for the sake of the example let's say it will be tradeId",
xml:" will contain the entire xml as a string"

}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 17, 2020, 7:35pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/6 "2020-08-17T19:35:09Z")

</div>

There is an example of an xml filter using xpath [here](https://discuss.elastic.co/t/help-logstash-xml-parsing-to-xpath/241051/2).

---

<div class="post-metadata">

**Author:** ![azer\_gorai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azer_gorai/32/74010_2.png) [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\
**Post date:** [August 17, 2020, 8:11pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/7 "2020-08-17T20:11:24Z")

</div>

i have previously worked with similar instructions , the thread you're referencing contains an error that i also encountered as well ( "Last 80 unconsumed characters:" )

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 17, 2020, 8:34pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/8 "2020-08-17T20:34:29Z")

</div>

That means your "XML" is not valid XML. There should be an exception logged telling you why. For example, in that thread it is ":exception=\>#\<REXML::ParseException: No close tag for /NewData".

If you XML is spread across multiple lines you will need to combine them into a single event. If you are using a file input a multiline codec may do the job. What does your input configuration look like?

---

<div class="post-metadata">

**Author:** ![azer\_gorai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azer_gorai/32/74010_2.png) [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\
**Post date:** [August 18, 2020, 2:38pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/9 "2020-08-18T14:38:50Z")

</div>

like this:

```auto
input {
  file {
    path => "C:/Users/user/Documents/shareCommon/pocc.xml"
    start_position => beginning
    sincedb_path => "nul"
    type => "xml"
    codec => multiline {

        pattern => "^<?FpML.*>"
        charset => "ISO-8859-1"
        negate => "true"
        what => "previous"
        max_lines => 100000
        
      }

  }
}
filter {
    xml {
        source => "message"
        target => "doc"
        store_xml => true
        xpath => ["/FpML/party/partyId/text()","fpml"]
        
    }

}

output {

  stdout {
    codec => rubydebug
  }
    
}  

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2020, 2:39pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269/10 "2020-09-15T14:39:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.