I'm experimenting incresing number of logstash crashing with the following error log:
JSON::GeneratorError: source sequence is illegal/malformed utf-8
to_json at json/ext/GeneratorMethods.java:71
to_json at /opt/logstash/lib/logstash/event.rb:148
receive at /opt/logstash/lib/logstash/outputs/redis.rb:158
handle at /opt/logstash/lib/logstash/outputs/base.rb:86
initialize at (eval):647
call at org/jruby/RubyProc.java:271
output at /opt/logstash/lib/logstash/pipeline.rb:266
outputworker at /opt/logstash/lib/logstash/pipeline.rb:225
start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152
Increasing the restarting cron rate of my logstash shippers, is useless, because once the log file appears to contain conflicting text (I guess some malformed text) logstash crashes... so no joy until end of conflicting log generation followed by logrotation (...a new clean log file, basically)
The problem seems to be related to some kind of UserAgent fields on logs at which bizarre strings are usually found. Most of the time everything goes smooth, but I feel the whole stack is vulnerable if no handling of this situation can be done:
A single computer can connect to all my servers, write a conflicting logline on all fo them, and let all logstash instances crash...
The issue is not new... there are entries about this googling around from 2012 or so...
I'm running latest 1.4.5 logstash and overall it goes well...
Trying to set charset at the input (as suggested googling around) as proven useless for me... so its time to ask the gurus...
Has some one dealt with this sucessfully?
Thank you very much.