json {
source => "message"
target => "[@metadata][string-indexed]"
}
ruby {
code => "
# extract the string-indexed object to a local variable
string_indexed = event.get('[@metadata][string-indexed]')
# iterate over the keys in the order of the string-encoded integer index
items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
# get the item from the indexed map
string_indexed[key]
end
# set the items array.
event.set('[@metadata][items-array]', items_array)
"
}
split {
field => "[@metadata][items-array]"
target => "[@metadata][single-item]"
}
mutate {
rename => {
"[@metadata][single-item][sensorTime]" => "sensorTime"
"[@metadata][single-item][sensorValue]" => "sensorValue"
"[@metadata][single-item][sensorId]" => "sensorId"
}
}
}
I would like to create the following event format:
one document per XXX.YY with the following fields:
Do not use @metadata until you are sure the fields have the form you want. Use
output { stdout { codec => rubydebug } }
and make sure the fields look the way you need them to look. Once you are sure that, for example, [metadata][string-indexed] looks the way you want, change the references to be [@metadata][string-indexed]
They currently do not. For example, you want the data sub-object when fetching string_indexed
and the first array entry when iterating over the array
string_indexed[key][0]
Lastly, you cannot use [@metadata] as the target of the split. (Or at least I cannot get it to work. rubydebug { metadata => true } shows the fields exists but the mutate+rename all return nil.) Try this:
Extracting the first two fields of the sensorId and adding them to the field names (which sounds like a bad idea to me) is left as an exercise for the reader. I know I have written examples in this forum of iterating over the event hash and renaming fields.
json {
source => "message"
target => "[metadata][string-indexed]"
}
ruby {
code => "
# extract the string-indexed object to a local variable
string_indexed = event.get('[metadata][string-indexed][data]')
# iterate over the keys in the order of the string-encoded integer index
items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
# get the item from the indexed map
string_indexed[key]
end
# set the items array.
event.set('[metadata][items-array]', items_array)
"
}
split {
field => "[metadata][items-array]"
target => "[metadata][single-item]"
}
mutate {
rename => {
"[metadata][single-item][sensorTime]" => "sensorTime"
"[metadata][single-item][sensorValue]" => "sensorValue"
"[metadata][single-item][sensorId]" => "sensorId"
}
remove_field => "[metadata][string-indexed]"
remove_field => "[metadata][items-array]"
}
My mutate function does not work, because [metadata][single-item] is an array containing 1 entry [0] ?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.