Xml filter with keyvalue pair

Elastic Stack Logstash
1

Hi,

I have an xml input like this, which has a keyValue, containing , and .
The multiple in caused logstash to have data type mismatch.
Is there a way in the xml filter to map as the field name and as the value or parse everything in xml filter to string? Thanks.

xml input:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header>
      ...
   </soap:Header>
   <soap:Body>
      <ns2:GetFormsSetResponse xmlns:ns2="http://xmlns.bmogc.net/com/bmo/ebs/eforms/sdss/V2" xmlns:ns3="http://xmlns.bmogc.net/bmo/2002/header/" xmlns:ns4="http://xmlns.bmogc.net/hub/header/V1">
         <header>
            <correlationId>4faaa6a3-9fd6-4b81-bf69-633ea713c821</correlationId>
         </header>
         <packages>
            <processControl>
               <transitNumber>1039</transitNumber>
               <sessionCustomer>
                  <customerID>685424209891550</customerID>
                  <name>
                     <firstName>GLSITABCD13</firstName>
                     <lastName>GLFABCD13</lastName>
                  </name>
               </sessionCustomer>
               <language>en-CA</language>
            </processControl>
            <package>
               <packageObjectStore>CanadianPnC</packageObjectStore>
               <packageType>CustomerPackage</packageType>
               <packageIndex>1</packageIndex>
               <deliveryDestinations>
                  <deliveryDestination>
                     <type>EFORM</type>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>FILENET</type>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>DTS</type>
                     <keyValues>
                        <keyValue>
                           <key>AutoClose</key>
                           <dataType>boolean</dataType>
                           <value>false</value>
                        </keyValue>
                        <keyValue>
                           <key>PendingReferenceExpiration</key>
                           <dataType>datetime</dataType>
                           <value>2018-11-20T14:15:41.973-05:00</value>
                        </keyValue>
                        <keyValue>
                           <key>CaptureMethod</key>
                           <dataType>string</dataType>
                           <value>BATCH SCANNED</value>
                        </keyValue>
                     </keyValues>
                  </deliveryDestination>
                  <deliveryDestination>
                     <type>PRINTER</type>
                     <keyValues>
                        <keyValue>
                           <key>location</key>
                           <dataType>string</dataType>
                           <value>\\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4</value>
                        </keyValue>
                     </keyValues>
                  </deliveryDestination>
               </deliveryDestinations>

Error:

[2018-11-20T21:26:02,278][WARN ][logstash.outputs.elasticsearch] eGroup>...</soap:Envelope>" /tmp/temp], :response=>{"index"=>{"_index"=>"dts-2018.11.20", "_type"=>"log", "_id"=>"AWczBANMTH81EwFHiK-Y", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [SOAP.Body.GetFormsSetResponse.packages.package.deliveryDestinations.deliveryDestination.keyValues.keyValue.value] of different type, current_type [date], merged_type [text]"}}}}

Regards,
Perry

2

hope this helps you,

3

Thanks. But is there a way to dynamically map KeyValue into fields? That xml is just an example, I dont know what the exact fields are.
e.g.

            <keyValue>
               <key>location</key>
               <dataType>string</dataType>
               <value>\\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4</value>
            </keyValue>

Expected to create field {{path_from_xml}}.location with value \\D2971A04QD005.percomqa.adrootqa.bmogc.net\Printer4.

4

Hi,

I tried below filter but still it's not working (it's showing the same error).
Seems like the xPath did not force the element to be stored seperately, but instead was stored in a new field and the original field persists.
Would you please advise how should I tell xml filter to always parse fields as string? Thanks.

code:

      xml {
         source => "Detail"
         target => "SOAP"
         xpath => ["//keyValue/value/text()", "keyValue_value",
                   "//keyValue/key/text()", "keyValue_key"]
      }
      ruby {
        code => "
          v = event.get('keyValue_value')
          k = event.get('keyValue_key')
          carr = []
          k.each_index { |i|
             h = { 'keyValue_value' => v[i] , 'keyValue_key' => k[i]}
             carr << h
           }
          event.set('SOAPitem', carr) "
      }

Error:

[2018-11-21T17:01:55,323][WARN ][logstash.outputs.elasticsearch] ... ], :response=>{"index"=>{"_index"=>"dts-2018.11.21", "_type"=>"log", "_id"=>"AWc3OJbBoll3OVKp9-um", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [SOAP.Body.GetFormsSetResponse.packages.package.deliveryDestinations.deliveryDestination.keyValues.keyValue.value] of different type, current_type [date], merged_type [text]"}}}}

5

If you do not specify the xpaths, you should get fields populated with strings or arrays of strings when multiple nodes exist with the same name. When you specify xpaths with //, your basically telling the filter to do what it was already going to do and combine all matching node values.

Forgive me, I don't really know ruby, but if you are trying to create a separate event for each /keyvalue node, they'll have to be ingested as separate lines. Furthermore, based on the error, the Elasticsearch is expecting date type data but it's getting text type data. Seems like you just need to update the field mapping type and it will work.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.