Mutate Convert XML Multiple Layers

Elastic Stack Logstash
1

I'm running into an issue trying to use mutate to convert and XML file. I can convert the first level, but if the XML file has multiple elements then they output as string.

Logstash Configuration

filter {
    if [fields][log] == "INSPECTION_AOI" {
        xml {
            source => "message"
            store_xml => true
            target => "Inspection"
            remove_field => [
                "message"
            ]
        }
        mutate {
            convert => {
                "[Inspection][CycleTime]" => "integer"
                "[Inspection][TotalFeatures]" => "integer"
                "[Inspection][TotalComponents]" => "integer"
                "[Inspection][LotSize]" => "integer"
                "[Inspection][Board][0][TotalFeatures][Value]" => "integer"
                "[Inspection][Board][0][TotalComponents][Value]" => "integer"
                "[Inspection][Board][0][Component][0][IncludedInCurrentVariant]" => "boolean"
                "[Inspection][Board][0][Component][0][TotalFeatures]" => "integer"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][X][0][Value]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][X][0][Minimum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][X][0][Maximum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][X][0][Target]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][Y][0][Value]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][Y][0][Minimum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][Y][0][Maximum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][Y][0][Target]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][ConfidenceLevel][0][Value]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][ConfidenceLevel][0][Minimum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][ConfidenceLevel][0][Maximum]" => "float"
                "[Inspection][Board][0][Component][0][Feature][0][Inspection][0][Measurements][0][ConfidenceLevel][0][Target]" => "float"
          }
        }
    }
}

output {
    if [fields][log] == "INSPECTION_AOI" {
        elasticsearch {
            hosts => "localhost"
            index => "inspection_aoi"
        }
        
        stdout { codec => rubydebug }
    }
}

File 1 (XML)

<Panel Id="1" Name="Panel" PanelStatus="Failed" StartTime="04/10/2019 09:09:07" EndTime="04/10/2019 09:11:12" CycleTime="4" TotalFeatures="7704" TotalComponents="7704" LotCode="TEST" LotSize="0" Barcode="10695 05 A07502 4718" Code="10695 05 A07502 4718" NG_Stacker_SlotNumber="">
  <Board Id="1" Name="Board1" BoardStatus="Failed" TotalFeatures="7704" TotalComponents="7704" Barcode="" Skip="false">
    <Component Id="6" Name="C1_D1" PartNumber="LED" Package="LED" Barcode="" IncludedInCurrentVariant="false" ComponentStatus="Passed" TotalFeatures="1">
      <Feature Id="7" Identifier="Body" FeatureStatus="Passed" FeatureInspectionInfo="" MeasurementValid="">
          <Inspection Identifier="Appearance" Type="AI2" status="Passed">
            <Measurements>
              <X Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <Y Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <ConfidenceLevel Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
            </Measurements>
          </Inspection>
      </Feature>
    </Component>
  </Board>
</Panel>

File 2 (XML)

<Panel Id="1" Name="Panel" PanelStatus="Failed" StartTime="04/10/2019 09:09:07" EndTime="04/10/2019 09:11:12" CycleTime="4" TotalFeatures="7704" TotalComponents="7704" LotCode="TEST" LotSize="0" Barcode="10695 05 A07502 4718" Code="10695 05 A07502 4718" NG_Stacker_SlotNumber="">
  <Board Id="1" Name="Board1" BoardStatus="Failed" TotalFeatures="7704" TotalComponents="7704" Barcode="" Skip="false">
    <Component Id="6" Name="C1_D1" PartNumber="LED" Package="LED" Barcode="" IncludedInCurrentVariant="false" ComponentStatus="Passed" TotalFeatures="1">
      <Feature Id="7" Identifier="Body" FeatureStatus="Passed" FeatureInspectionInfo="" MeasurementValid="">
          <Inspection Identifier="Appearance" Type="AI2" status="Passed">
            <Measurements>
              <X Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <Y Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <ConfidenceLevel Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
            </Measurements>
          </Inspection>
          <Inspection Identifier="Appearance" Type="AI2" status="Passed">
            <Measurements>
              <X Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <Y Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <ConfidenceLevel Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
            </Measurements>
          </Inspection>
          <Inspection Identifier="Appearance" Type="AI2" status="Passed">
            <Measurements>
              <X Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <Y Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
              <ConfidenceLevel Value="-0.006" Minimum="-0.079999999999886329" Maximum="0.0800000000001137" Target="0" />
            </Measurements>
          </Inspection>
      </Feature>
    </Component>
  </Board>
</Panel>

Result
File 1 - ConfidenceLevel, Y, X come in a FLOAT for index 0

File 2 - ConfidenceLevel, Y, X come in a FLOAT for index 0
File 2 - ConfidenceLevel, Y, X come in a STRING for index 1
File 2 - ConfidenceLevel, Y, X come in a STRING for index 2

I realize I'm only converting the first index of the XML in my logstash configuration. Is there a way to define a dynamic value for the index instead of hardcoding a [0].

2

I was going to suggest you use the 'force_array => false' option on the xml filter. Then the inner Inspection field will be an array but Board, Component, and Feature will not. However I now realize that is a mistake, since it makes the first XML harder to parse.

You would use a ruby filter to handle the inner Inspection. This is an example of how to convert the second XML

    xml {
        source => "message"
        store_xml => true
        target => "Inspection"
        remove_field => [ "message" ]
        force_array => false
    }
    ruby {
        code => '
            replacementInspection = []
            i = event.get("[Inspection][Board][Component][Feature][Inspection]")
            i.each_index { |x|
                replacementMeasurements = {}
                r = i[x]
                m = r["Measurements"]
                m.each { |k, v|
                    h = {}
                    h["Value"] = v["Value"].to_f
                    h["Maximum"] = v["Maximum"].to_f
                    h["Minimum"] = v["Minimum"].to_f
                    h["Target"] = v["Target"].to_f
                    replacementMeasurements[k] = h
                }
                r["Measurements"] = replacementMeasurements
                replacementInspection << r
            }
            event.set("[Inspection][Board][Component][Feature][Inspection]", replacementInspection)
        '
    }

It needs to be re-written to handle 'force_array => true' and then handle the first XML format But if you can do ruby (and if you do a lot of logstash it is worth learning) then it should help you.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.