Logstash is not processing incoming events

[jakubhajek]

Hello,

We are using Logstash 5.2.0 connected with Elasticsearch. The application is sending the logs in XML via HTTP input format which than is parsed by XML filter to JSON document. The Logstash is configured to use default memory queue settings. In some circumstances, XML sent from the application receives an incorrect syntax causing warning for XML Simple module.
The following warning is logged:

[2017-08-25T11:15:37,049][WARN ][logstash.filters.xml ] Error parsing xml with XmlSimple {:source=>"message", :value=>"<xml>

Just after that we receive the following error:

[2017-08-25T11:15:37,057][ERROR][logstash.pipeline ] (CachingCallSite.java:177)

Logstash is going to process the XML document which doesn’t have the correct syntax and following errors and warnings are thrown:

[2017-08-25T11:15:37,332][logstash.filters.xml     ] Error parsing xml with XmlSimple {:source=>"message", :value=>"<xml>
[2017-08-25T11:15:37,336][ERROR][logstash.pipeline        ] hingCallSite.java:188)

[2017-08-25T11:15:37,546][logstash.filters.xml     ] Error parsing xml with XmlSimple {:source=>"message", :value=>"<xml>
[2017-08-25T11:15:37,550][ERROR][logstash.pipeline        ] (CachingCallSite.java:177)", "org.jruby.run

When this occurs Logstash is not able to process other incoming events from the application and all incoming requests sent via HTTP are stalled. The only way to unblock that is to restart Logstash. Once the restart process is initiated, the following error is thrown.

[2017-08-25T13:05:47,235][WARN ][logstash.runner          ] SIGTERM received. Shutting down the agent.
[2017-08-25T13:05:47,241][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-08-25T13:05:52,248][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>8, "stalling_thread_info"=>{"other"=>[{"thread_id"=>30, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.23-java/lib/logstash/inputs/beats.rb:206:in `run'"}, {"thread_id"=>31, "name"=>"[main]<http", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-input-http-3.0.5/lib/logstash/inputs/http.rb:168:in `join'"}]}}
[2017-08-25T13:05:52,250][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

We have tried to solve that problem by skipping the document by adding following code to the config.

ruby {
    code => "
        tags = event.get('[tags]')
        if tags.instance_of?(Array)
            tags_string = tags.join
            if tags_string =~ /failure|exception/
                event.cancel
            end
        end
    "
}

but it doesn't solve the issue, the same error occured as it has been described.

Thank you,