Create nested field with XPath


#1

I have an XML structure

<Event>
  <Core Id="10233" />
  <Parameters>
      <Parameter EngValue="1.0" DecValue="1.0" />
      <Parameter EngValue="GCOM" />
      <Parameter EngValue="1.0" DecValue="1.0" />
  </Parameters>
</Event>

I'd like logstash to output this structure :

{
  "CoreID" : "10233",
  "Parameter" : [
    {
      "EngValue" : "1.0",
      "DecValue" : "1.0"
    },
    {
      "EngValue" : "GCOM"
    },
    {
      "EngValue" : "1.0",
      "DecValue" : "1.0"
    }
  ]
}

I have this mapping in ES :

"mappings" : {
  "logs" : {
    "properties" : {
      "CoreId" :{
        "type" : "text"
      },
      "Parameter" : {
        "type" : "nested",
        "properties" : {
          "DecValue" : {
            "type" : "float"
          },
          "EngValue" : {
            "type" : "text"
          }
        }
      }
    }
  }
} 

With the XML filter I tried :

xpath => [
    "/Event/Core/@Id", "CoreID",
    "/Event/Parameters/Parameter/@DecValue", "[Parameter][DecValue]",
    "/Event/Parameters/Parameter/@EngValue", "[Parameter][EngValue]",
]

But so far I only got this :

{
  "CoreID" : "10233",
  "Parameter" : {
      "EngValue" : ["1.0", "GCOM", "1.0"],
      "DecValue" : ["1.0", "1.0"]
    }
}

Any idea how I can achieve this?
How can I make "Parameter" the array instead of its attributes?


(Magnus Bäck) #2

It appears you're asking the same question as here:


#3

I almost have the same problem.
But I'm not sure that this solution could work for me :


Right after the XML filter my fields look like this :

"Parameter" : {
   "EngValue" : ["1.0", "GCOM", "1.0"],
   "DecValue" : ["1.0", "1.0"]
 }

And I can't just join the elements of each array : For each element of "EngValue", I don't know the corresponding element in "DecValue" (since "DecValue" is optional).
I could do it if I had this after the XML filter :

"Parameter" : {
   "EngValue" : ["1.0", "GCOM", "1.0"],
   "DecValue" : ["1.0",   ""  , "1.0"]
 }

Please notice that I don't want to split my values in multiple events. I want to put them in a "Parameter" array ("Parameter" is a nested object).


#4

The solution was to use a ruby filter instead of XPath :

filter {
  xml {
    source => "message"
    target => "parsed"
  }

  ruby {
      code => '
        event.set("Parameter", event.get("[parsed][Parameters][0][Parameter]"))
        event.set("CoreId", event.get("[parsed][Core][0][Id]"))
      '
  }


  mutate {
    remove_field => ["message", "parsed"]
  }
}

This answer comes from Stack overflow : https://stackoverflow.com/questions/45938870/handle-nested-object-with-logstah-and-xpath


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.