That's blowing up here. Per this comment on the same error in a different context it means you have non-UTF-8 characters in one of your fields.
OK, if in a ruby filter you do
text = [0xD1].pack("C*")
text.to_json
you will get
"\xD1" from ASCII-8BIT to UTF-8 {:class=>"Encoding::UndefinedConversionError"
You have a couple of options. If you know the encoding of the string you have you
could try this ruby code
text = event.get("problemField")
text = text.force_encoding("iso-8859-1").encode("utf-8")
event.set("problemField", text)
which will get you
"problemField" => "Ñ",
I am not telling you that you have iso-8859-1 encoded text. It's plausible but you need to determine if this is true.
If the encoding is unknown or varies then a more aggressive approach would be
event.set("problemField", text.encode("UTF-8", "binary", :invalid => :replace, :undef => :replace)
which will get you
"problemField" => "�",
I recognize that losing data is bad, but your fields have to be valid UTF-8 to be sent to elasticsearch. It is not optional.
I assume you know which fields are likely to have non-UTF-8 data in them. If you do not know then you will have to iterate over the fields of the event. This code should give you some ideas.
I hope this helps you to understand the issue.
Another thing you could do in ruby is
initialEvent = event.to_json # String.to_json crashes, Hash.to_json does not
fixedEvent = initialEvent.encode("UTF-8", "binary", "replace" => "x", :invalid => :replace, :undef => :replace)
if fixedEvent != initialEvent
event.tag("encodingProblem")
end
then route to a different output based on "encodingProblem" in [tags], if you use a rubydebug output then start looking for \x, as in
"problemField" => "\xD1",
"tags" => [
[0] "encodingProblem"
],