# Xml filter with keyvalue pair

**URL:** <https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613>\
**Category:** Logstash\
**Created:** [November 20, 2018, 10:22pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613 "2018-11-20T22:22:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![perryparktung](https://avatars.discourse-cdn.com/v4/letter/p/d78d45/32.png) [@perryparktung](https://discuss.elastic.co/u/perryparktung)\
**Post date:** [November 20, 2018, 10:22pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/1 "2018-11-20T22:22:53Z")

</div>

Hi,

I have an xml input like this, which has a keyValue, containing , and .  
The multiple in caused logstash to have data type mismatch.  
Is there a way in the xml filter to map as the field name and as the value or parse everything in xml filter to string? Thanks.

xml input:

```
<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header>
      ...
   </soap:Header>
   <soap:Body>
      <ns2:GetFormsSetResponse xmlns:ns2="http://xmlns.bmogc.net/com/bmo/ebs/eforms/sdss/V2" xmlns:ns3="http://xmlns.bmogc.net/bmo/2002/header/" xmlns:ns4="http://xmlns.bmogc.net/hub/header/V1">
         <header>
            <correlationId>4faaa6a3-9fd6-4b81-bf69-633ea713c821</correlationId>
         </header>
         <packages>
            <processControl>
               <transitNumber>1039</transitNumber>
               <sessionCustomer>
                  <customerID>685424209891550</customerID>
                  <name>
                     <firstName>GLSITABCD13</firstName>
                     <lastName>GLFABCD13</lastName>
                  </name>
               </sessionCustomer>
               <language>en-CA</language>
            </processControl>
            <package>
               <packageObjectStore>CanadianPnC</packageObjectStore>
               <packageType>CustomerPackage</packageType>
               <packageIndex>1</packageIndex>
               <deliveryDestinations>
                  <deliveryDestination>
                     <type>EFORM</type>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>FILENET</type>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>DTS</type>
                     <keyValues>
                        <keyValue>
                           <key>AutoClose</key>
                           <dataType>boolean</dataType>
                           <value>false</value>
                        </keyValue>
                        <keyValue>
                           <key>PendingReferenceExpiration</key>
                           <dataType>datetime</dataType>
                           <value>2018-11-20T14:15:41.973-05:00</value>
                        </keyValue>
                        <keyValue>
                           <key>CaptureMethod</key>
                           <dataType>string</dataType>
                           <value>BATCH SCANNED</value>
                        </keyValue>
                     </keyValues>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>PRINTER</type>
                     <keyValues>
                        <keyValue>
                           <key>location</key>
                           <dataType>string</dataType>
                           <value>\\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4</value>
                        </keyValue>
                     </keyValues>
                  </deliveryDestination>
               </deliveryDestinations>

```

Error:

`[2018-11-20T21:26:02,278][WARN][logstash.outputs.elasticsearch] eGroup>...</soap:Envelope>" /tmp/temp], :response=>{"index"=>{"_index"=>"dts-2018.11.20", "_type"=>"log", "_id"=>"AWczBANMTH81EwFHiK-Y", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [SOAP.Body.GetFormsSetResponse.packages.package.deliveryDestinations.deliveryDestination.keyValues.keyValue.value] of different type, current_type [date], merged_type [text]"}}}}`

Regards,  
Perry

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 21, 2018, 6:11am UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/2 "2018-11-21T06:11:21Z")

</div>

hope this helps you,

> [@Is it possible to change output structure](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/17):
>
> The input code as per your requirement is as follows, input { file { path =\> "D:/xxxxx/ELKStack/sample.xml" start\_position =\> "beginning" sincedb\_path =\> "NUL" codec =\> multiline { pattern =\> "" negate =\> "true" what =\> "previous" auto\_flush\_interval =\> 1 max\_lines =\> 333333 } } } filter { xml { source =\> "message" target =\> "parsed" store\_xml =\> "false" xpath =\> [ "/system/Report/ReportHost/@ip","ip", "/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname", …

---

<div class="post-metadata">

**Author:** ![perryparktung](https://avatars.discourse-cdn.com/v4/letter/p/d78d45/32.png) [@perryparktung](https://discuss.elastic.co/u/perryparktung)\
**Post date:** [November 21, 2018, 2:41pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/3 "2018-11-21T14:41:07Z")

</div>

Thanks. But is there a way to dynamically map KeyValue into fields? That xml is just an example, I dont know what the exact fields are.  
e.g.

> ```
> <keyValue>
> <key>location</key>
> <dataType>string</dataType>
> <value>\\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4</value>
> </keyValue>
> 
> ```

Expected to create field `{{path_from_xml}}.location` with value `\\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4`.

---

<div class="post-metadata">

**Author:** ![perryparktung](https://avatars.discourse-cdn.com/v4/letter/p/d78d45/32.png) [@perryparktung](https://discuss.elastic.co/u/perryparktung)\
**Post date:** [November 21, 2018, 5:09pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/4 "2018-11-21T17:09:47Z")

</div>

Hi,

I tried below filter but still it's not working (it's showing the same error).  
Seems like the xPath did not force the element to be stored seperately, but instead was stored in a new field and the original field persists.  
Would you please advise how should I tell xml filter to always parse fields as string? Thanks.

code:

```
      xml {
         source => "Detail"
         target => "SOAP"
         xpath => ["//keyValue/value/text()", "keyValue_value",
                   "//keyValue/key/text()", "keyValue_key"]
      }
      ruby {
        code => "
          v = event.get('keyValue_value')
          k = event.get('keyValue_key')
          carr = []
          k.each_index { |i|
             h = { 'keyValue_value' => v[i] , 'keyValue_key' => k[i]}
             carr << h
           }
          event.set('SOAPitem', carr) "
      }

```

Error:

`[2018-11-21T17:01:55,323][WARN][logstash.outputs.elasticsearch] ... ], :response=>{"index"=>{"_index"=>"dts-2018.11.21", "_type"=>"log", "_id"=>"AWc3OJbBoll3OVKp9-um", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [SOAP.Body.GetFormsSetResponse.packages.package.deliveryDestinations.deliveryDestination.keyValues.keyValue.value] of different type, current_type [date], merged_type [text]"}}}}`

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [November 26, 2018, 4:50pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/5 "2018-11-26T16:50:55Z")

</div>

If you do not specify [the xpaths](https://www.elastic.co/guide/en/logstash/6.4/plugins-filters-xml.html#plugins-filters-xml-xpath), you should get fields populated with strings or arrays of strings when multiple nodes exist with the same name. When you specify [xpaths with `//`](https://www.w3schools.com/xml/xpath_syntax.asp), your basically telling the filter to do what it was already going to do and combine all matching node values.

Forgive me, I don't really know ruby, but if you are trying to create a separate event for each /keyvalue node, they'll have to be ingested as separate lines. Furthermore, based on the error, the Elasticsearch is expecting [date type data](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/date.html) but it's getting [text type data](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/text.html). Seems like you just need to update the field mapping type and it will work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 24, 2018, 4:50pm UTC](https://discuss.elastic.co/t/xml-filter-with-keyvalue-pair/157613/6 "2018-12-24T16:50:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.