XML key:value Extraction

Elastic Stack Logstash
1

Apologies, I have searched but cannot seem to find a good example on how to extract and then index the remaining data. If this is addressed elsewhere and Ive missed it please direct me there. Being new to logstash and its filtering mechanism I cannot seem to find how to take the following xml file format:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bclog="http://www.ciglo.com/wcf/log/v1_0" elementFormDefault="unqualified" targetNamespace="http://www.ciglo.com/wcf/log/v1_0" version="1.0">
  <xs:element name="bcLogEntry" type="bclog:bcLogEntry"/>

      <xs:complexType name="bcLogEntry">
        <xs:sequence>
          <xs:element minOccurs="0" name="c_ip" type="xs:string">
            <xs:annotation>
              <xs:documentation><![CDATA[abrigo: c-ip]]></xs:documentation>
            </xs:annotation>
          </xs:element>
          <xs:element minOccurs="0" name="c_port" type="xs:int">
            <xs:annotation>
              <xs:documentation><![CDATA[abrigo: c-port]]></xs:documentation>
            </xs:annotation>
          </xs:element>
          <xs:element minOccurs="0" name="cs_Accept_" type="xs:string">
            <xs:annotation>
              <xs:documentation><![CDATA[abrigo: cs(Accept)]]>

and only index the fields as:

name:c_ip, type:string
name:c_port, type:int
name:cs_Accept_" type=string

etc.

I would be grateful for any assistance. Thank you.

2

Please edit your post, select the XML (not the whole post) and click on </> in the toolbar above the edit pane.

3

Assuming you ingest a complete and valid XML string as a single event, then the following filter

    xml { source => "message" target => "[@metadata][theXML]" }
    ruby {
        code => '
            elements = event.get("[@metadata][theXML][complexType][0][sequence][0][element]")
            elements.each { |x|
                event.set(x["name"], x["type"])
            }
        '
    }

will get you an event with

    "c_port" => "xs:int",
      "c_ip" => "xs:string",
"cs_Accept_" => "xs:string",

which is probably not quite what you want, but should give you an idea of how to get there.

4

Badger,
That's awesome. Thank you very much. That definitely gave me direction.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.