We frequently have documents fail due to encoding issues.
A typical error message is
{:timestamp=>"2017-03-03T09:04:54.473000+0000", :message=>"Received an event that has a different character encoding than you configured.", :text=>"2017-03-03T09:04:54.404238+00:00 m4ud3.prd.mel.mmd m4ud[4070]: INFO 10.16.56.202 - sent message from XXXXXX to xxxxxxxxx; account=\"XXXXX\" customerId=\"XXXXX\" previousTimeout=300 to=\"xxxxxxxx\" message=\", <BUSINESS_NAME=In > erinnert an , 4. M\xE4rz 2017. <CONFIRM_TEXT=Termin best\xE4tigen><CONFIRM_DATETIME= Samstag, 4. M\xE4rz 2017><CONFIRM_RESP=Ja>=Best\xE4tigen, <CANCEL_TEXT=Termin absagen><CANCEL_RESP=Nein>=Absagen \\r<HTML=Bitte klicken Sie, um den Termin zu best\xE4tige an ><PLAINTEXT=Mit Ja best\xE4tigen, mit Nenrufen >Odten \" messageId=\"392\"", :expected_charset=>"UTF-8", :level=>:warn}
(i've removed much of the plain text for sake of confidentiality)
This is pulled from an rsyslog file using the rsyslog plugin. We deal with a wide range of encodings so want to keep the content as is, but if we cant, I'd rather drop the offending field than have it fail completely?
What's the best way to approach this?