Is there a way to setup a pipeline to run auto convert on all sub-fields of a container?
Basically I would like to do something like (had wildcards been supported):
I found one way to do this through logstash and ruby plugins. Posting here in case anyone is in the same situation. In my specific case I use a kv filter to get the values I need. I therefore ended up doing a loop through the output of the kv filter, converting everything to integer if possible, float if that's not possible and just leaving it as is otherwise. (I probably don't need the a[k] = a[k], but I'm not used to Ruby, so just put it there in case Ruby doesn't like rescue to b empty)
kv {
target => "log_params"
}
ruby {
code => '
a = event.get("log_params")
a.each { |k, v|
begin
a[k] = Integer(a[k])
rescue
begin
a[k] = Float(a[k])
rescue
a[k] = a[k]
end
end
}
event.set("log_params", a)
'
}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.