[JSON filter] Split Dynamic fields

[manunc]

Hello, I try to create an event for each fields named "XXX.YY" coming from the following JSON message (data object):

{"data":{
"316.2":[{"sensorId":"316.2.3","sensorTime":1549599391,"sensorValue":0.08}],
"315.10":[{"sensorId":"315.10.8","sensorTime":1549599019,"sensorValue":43.01}],
"314.6":[{"sensorId":"314.6.3","sensorTime":1549599083,"sensorValue":0.76}],
"314.9":[{"sensorId":"314.9.3","sensorTime":1549599019,"sensorValue":0.606}],
"315.1":[{"sensorId":"315.1.3","sensorTime":1549599321,"sensorValue":0.94}],
"313.4":[{"sensorId":"313.4.3","sensorTime":1549599576,"sensorValue":0.52}]
}}

My filter I try to adapt:

	json {
	    source => "message"
	    target => "[@metadata][string-indexed]"
	}
	
	ruby {
	    code => "
	      # extract the string-indexed object to a local variable
	      string_indexed = event.get('[@metadata][string-indexed]')
	      
	      # iterate over the keys in the order of the string-encoded integer index 
	      items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
	      	# get the item from the indexed map
	      	string_indexed[key]
	      end
	      
	      # set the items array.
	      event.set('[@metadata][items-array]', items_array)
	    "
	}
	
	split {
	    field => "[@metadata][items-array]"
	    target => "[@metadata][single-item]"
	}
	
	mutate {
	    rename => {
	      "[@metadata][single-item][sensorTime]" 	=> "sensorTime"
	      "[@metadata][single-item][sensorValue]"	=> "sensorValue"
	      "[@metadata][single-item][sensorId]"	=> "sensorId"
	    }
	}
	  
	}

I would like to create the following event format:
one document per XXX.YY with the following fields:

data.316.2.sensorTime
data.316.2.sensorId
data.316.2.sensorValue

[Badger]

Do not use @metadata until you are sure the fields have the form you want. Use

output { stdout { codec => rubydebug } }

and make sure the fields look the way you need them to look. Once you are sure that, for example, [metadata][string-indexed] looks the way you want, change the references to be [@metadata][string-indexed]

They currently do not. For example, you want the data sub-object when fetching string_indexed

string_indexed = event.get('[@metadata][string-indexed][data]')

and the first array entry when iterating over the array

 string_indexed[key][0]

Lastly, you cannot use [@metadata] as the target of the split. (Or at least I cannot get it to work. rubydebug { metadata => true } shows the fields exists but the mutate+rename all return nil.) Try this:

    split {
        field => "[@metadata][items-array]"
        target => "[metadata][single-item]"
    }
    mutate {
        rename => {
          "[metadata][single-item][sensorTime]"        => "sensorTime"
          "[metadata][single-item][sensorValue]"       => "sensorValue"
          "[metadata][single-item][sensorId]"  => "sensorId"
        }
        remove_field => "[metadata]"
    }

That will get you events that look like

   "sensorId" => "316.2.3",
 "sensorTime" => 1549599391,
"sensorValue" => 0.08,

Extracting the first two fields of the sensorId and adding them to the field names (which sounds like a bad idea to me) is left as an exercise for the reader. I know I have written examples in this forum of iterating over the event hash and renaming fields.

[manunc]

Thanks, you help me a lot.

I succeed to get the following event now

  "metadata" => {
    "single-item" => [
        [0] {
             "sensorTime" => 1549867548,
            "sensorValue" => 0.52,
               "sensorId" => "316.2.3"
        }
    ]
},

using the following code:

	json {
	    source => "message"
	    target => "[metadata][string-indexed]"
	}
	
	ruby {
	    code => "
	      # extract the string-indexed object to a local variable
	      string_indexed = event.get('[metadata][string-indexed][data]')
	      
	      # iterate over the keys in the order of the string-encoded integer index 
	      items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
	      	# get the item from the indexed map
	      	string_indexed[key]
	      end
	      
	      # set the items array.
	      event.set('[metadata][items-array]', items_array)
	    "
	}
	
	split {
        field => "[metadata][items-array]"
        target => "[metadata][single-item]"
    }
    mutate {
        rename => {
          "[metadata][single-item][sensorTime]"     => "sensorTime"
          "[metadata][single-item][sensorValue]"    => "sensorValue"
          "[metadata][single-item][sensorId]"		=> "sensorId"
        }
        remove_field => "[metadata][string-indexed]"
        remove_field => "[metadata][items-array]"
    }

My mutate function does not work, because [metadata][single-item] is an array containing 1 entry [0] ?

[Badger]

As I said, when iterating over the array, you should reference the array entry using

string_indexed[key][0]