Catching JSON parse errors

Elastic Stack Logstash
1

Hi,

We have an old logstash system (v1.5) which had been running just fine until recently. Some of the messages are getting mangled, for reasons unknown, and the JSON filter keeps throwing parse fail errors. That causes logstash to panic and refuse new connections. This only happens intermittently.

While I investigate the reason for mangled logs, I'd like to ensure that logstash stops erroring out. Is there a way to catch JSON parse errors in logstash.

2

Please supply more details and logs from Logstash. The json filter should never panic over parse errors. It'll just tag failing events _jsonparsefailure and pass them on as plain text.

3 
{:timestamp=>"2017-03-10T09:01:22.302000+0000", :message=>"Trouble parsing json", :source=>"json_message", :raw=>"[ Invalid JSON ]", :exception=>#<LogStash::Json::ParserError: Unexpected character ('i' (code 105)): was expecting a colon to separate field name and value
 at [Source: [B@594c6d08; line: 1, column: 203]>, :level=>:warn}

After a bunch of these, I get logs like these

{:timestamp=>"2017-03-10T04:37:20.471000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2017-03-10T04:37:20.472000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-03-10T04:37:20.473000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2017-03-10T04:37:20.477000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2017-03-10T04:37:20.478000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-03-10T04:37:20.479000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
4

Okay, but I'm still not convinced that the JSON parse problem is what's causing the pipeline stall. The way I read the code invalid JSON will be passed through. Please show your Logstash configuration.

More recent versions of the filter have an option that silences the tagging and warning when parse errors occur.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.