Create nested field with XPath

[cadoman]

I have an XML structure

<Event>
  <Core Id="10233" />
  <Parameters>
      <Parameter EngValue="1.0" DecValue="1.0" />
      <Parameter EngValue="GCOM" />
      <Parameter EngValue="1.0" DecValue="1.0" />
  </Parameters>
</Event>

I'd like logstash to output this structure :

{
  "CoreID" : "10233",
  "Parameter" : [
    {
      "EngValue" : "1.0",
      "DecValue" : "1.0"
    },
    {
      "EngValue" : "GCOM"
    },
    {
      "EngValue" : "1.0",
      "DecValue" : "1.0"
    }
  ]
}

I have this mapping in ES :

"mappings" : {
  "logs" : {
    "properties" : {
      "CoreId" :{
        "type" : "text"
      },
      "Parameter" : {
        "type" : "nested",
        "properties" : {
          "DecValue" : {
            "type" : "float"
          },
          "EngValue" : {
            "type" : "text"
          }
        }
      }
    }
  }
} 

With the XML filter I tried :

xpath => [
    "/Event/Core/@Id", "CoreID",
    "/Event/Parameters/Parameter/@DecValue", "[Parameter][DecValue]",
    "/Event/Parameters/Parameter/@EngValue", "[Parameter][EngValue]",
]

But so far I only got this :

{
  "CoreID" : "10233",
  "Parameter" : {
      "EngValue" : ["1.0", "GCOM", "1.0"],
      "DecValue" : ["1.0", "1.0"]
    }
}

Any idea how I can achieve this?
How can I make "Parameter" the array instead of its attributes?

[magnusbaeck (Magnus Bäck)]

It appears you're asking the same question as here:

Xml parser with xpath giving wrong result in xml filter plugin

Hi All i am having a xml filter, which looks like this xml { source => "message" target => "message_parsed" add_tag => ["xml_parsed"] remove_namespaces => true store_xml => true force_array => true xpath => [ "/table/row/number/text()","number" ] } Here i am getting only 1 record. I am expecting 5 records Under the column "number" i am getting value as comma seperated 123,124,125,126,127 My expectation …

[cadoman]

I almost have the same problem.
But I'm not sure that this solution could work for me :

Use of Split Filter for more than 1 fields, it is possible

The split filter acts on a single field. You can use a ruby filter to join the elements of each array into another array so that you have this: { ... "field_to_split": [ ["A", "E", "W", "Q"], ... ] } Then feed field_to_split to a split filter, and use another filter or two to split the array back into discrete fields.

Right after the XML filter my fields look like this :

"Parameter" : {
   "EngValue" : ["1.0", "GCOM", "1.0"],
   "DecValue" : ["1.0", "1.0"]
 }

And I can't just join the elements of each array : For each element of "EngValue", I don't know the corresponding element in "DecValue" (since "DecValue" is optional).
I could do it if I had this after the XML filter :

"Parameter" : {
   "EngValue" : ["1.0", "GCOM", "1.0"],
   "DecValue" : ["1.0",   ""  , "1.0"]
 }

Please notice that I don't want to split my values in multiple events. I want to put them in a "Parameter" array ("Parameter" is a nested object).

[cadoman]

The solution was to use a ruby filter instead of XPath :

filter {
  xml {
    source => "message"
    target => "parsed"
  }

  ruby {
      code => '
        event.set("Parameter", event.get("[parsed][Parameters][0][Parameter]"))
        event.set("CoreId", event.get("[parsed][Core][0][Id]"))
      '
  }


  mutate {
    remove_field => ["message", "parsed"]
  }
}

This answer comes from Stack overflow : https://stackoverflow.com/questions/45938870/handle-nested-object-with-logstah-and-xpath