I configured it to use Split...but it looks like that's going to produce unfavorable results. The test record I've done it on only have one question, but it appears as though it will end up creating an array. Is it possible to set the target using split to something like target => "[question][%{questionid}]"? The intent would be to stick all the fields in a separate object under each question id...so like:
At least I was researching the right path, using ruby to manipulate the document. This is well and above anything I remotely understand with Ruby. Was I supposed to modify it, like the event.get from jdbcOutput to my actual field name, or just copy/paste? I copy/pasted it and nothing happens....no errors, no change in field structures.
Alright, that does the bulk of the work thank you. However, when splitting the fields out, they're all under question so the fields look like question.question2.integeranswer Is there a modification to this Ruby code to make it just question2.integeranswer or do I need to throw in an additional ruby script after this to modify them all?
That did it, I appreciate the assistance. I've said it before, I need to find time to learn ruby so I can do this kind of stuff myself...or at the very least understand what the hell is going on here.
Follow up question, can this ruby script be used to also remove the field if the value is null? I've tried using an additional ruby script I've used previously to remove empty fields, but it doesn't work on these fields, I'm assuming because they're nested.
ruby {
code => '
event.to_hash.each { |k, v|
if v.kind_of? String
if v == ""
event.remove(k)
end
end
}
'
}
That is correct. That ruby filter only checks the top-level fields. This is an example of ruby code that recursively descends into fields of an event and modifies them.
I'm very much lost in understanding most of this, but I THINK I changed it to remove null fields? Although, I think I'm missing something because I don't see where a target field is referencing...is it looking at the entire event or just the message field?
I'm not quite sure how to interpret what it's doing after the first couple logic checks. Define variable EmptyField, then if the object exists, and the object is a hash and not empty, then for each hash in the object do....something...
ruby {
code => '
def EmptyField(object, name, event)
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| EmptyField(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
EmptyField(object[i], "#{name}[#{i}]", event)
}
else
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length = "null"
event.remove(name)
end
end
end
end
event.to_hash.each { |k, v|
EmptyField(v, "[#{k}]", event)
}
'
}
Sorry, that is an assignment so it is unconditionally true. Try if object == "null". Note that null in JSON will be nil in Ruby, so you may need if ! object instead.
Tried if object == "null", if object == "nil", and if ! object and none of them worked. They didn't delete the event, but they didn't seem to do anything either.
Awesome thank you. This is where my inability to understand Ruby really shines, lol. Just one of the many things to add to the list of needing to give time to learn.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.