Logstash version: 2.3.4-1
OS: centos 7
config file:
input {
kafka {
bootstrap_servers => "${KAFKA_ADDRESS}"
topics => ["${LOGSTASH_KAFKA_TOPIC}"]
}
}
filter {
ruby {
code =>
"require 'json'
require 'base64'
.. some code in ruby ..
"
}
}
output {
http {
http_method => 'post'
url => '${APP_URL}/'
headers => ["AUTHORIZATION", "Basic ${CREDS}"]
}
stdout { }
}
Logstash sometimes complains it is dead, and stops processing. The error is not in logstash.err but in logstash.log
The errors look like
{:timestamp=>"2016-09-07T13:41:46.437000+0000", :message=>#<LogStash::PipelineReporter::Snapshot:0x6d7c925 @data={:events_filtered=>0, :events_consumed=>822, :worker_count=>1, :inflight_count=>0, :worker_states=>[{:status=>"dead", :alive=>false, :index=>0, :inflight_count=>0}], :output_info=>[{:type=>"http", :config=>{"http_method"=>"post", "url"=>"${APP_URL}/", "headers"=>["AUTHORIZATION", "Basic ${CREDS}"], "ALLOW_ENV"=>true}, :is_multi_worker=>false, :events_received=>0, :workers=><Java::JavaUtilConcurrent::CopyOnWriteArrayList:-392817559 [<LogStash::Outputs::Http http_method=>"post", url=>"", headers=>{..}, codec=><LogStash::Codecs::Plain charset=>"UTF-8">, workers=>1, request_timeout=>60, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automatic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, ssl_certificate_validation=>true, keystore_type=>"JKS", truststore_type=>"JKS", cookies=>true, verify_ssl=>true, format=>"json">]>, :busy_workers=>1}, {:type=>"stdout", :config=>{"ALLOW_ENV"=>true}, :is_multi_worker=>false, :events_received=>0, :workers=><Java::JavaUtilConcurrent::CopyOnWriteArrayList:-1413712692 [<LogStash::Outputs::Stdout codec=><LogStash::Codecs::Line charset=>"UTF-8", delimiter=>"\n">, workers=>1>]>, :busy_workers=>0}], :thread_info=>, :stalling_threads_info=>}>, :level=>:warn}
Looking at the code, this seems that the threads died for some reason, but something keeps writing to the log that it is dead.
What does this error mean, and is there a way for it to not happen, or at least, to have it restart automatically if it does happen?