I wrote a simple sql query to get test data it pulls the top 100 with no problem when I tried to remove the top 100 and pull in the 2500 records in the table logstash shutsdown after a few seconds with the following from the error log:
[2017-05-09T14:10:03,376][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::HTTP:0x3fe240a URL:http://localhost:9200>]}
[2017-05-09T14:10:03,376][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2000}
[2017-05-09T14:10:04,078][INFO ][logstash.pipeline ] Pipeline main started
[2017-05-09T14:10:04,141][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-09T14:10:04,172][INFO ][logstash.inputs.jdbc ] (0.062000s) select Employee_id,last_name,First_Name,title,DepartmentID from dbo.employee
[2017-05-09T14:10:09,164][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-05-09T14:10:14,203][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1071, "stalling_thread_info"=>{"other"=>[{"thread_id"=>30, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in synchronize'"}, {"thread_id"=>31, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:insynchronize'"}, {"thread_id"=>34, "name"=>"[main]>worker4", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/awesome_print-1.7.0/lib/awesome_print/inspector.rb:126:in tty?'"}, {"thread_id"=>35, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:insynchronize'"}, {"thread_id"=>36, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in synchronize'"}, {"thread_id"=>38, "name"=>"[main]>worker8", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:insynchronize'"}, {"thread_id"=>39, "name"=>"[main]>worker9", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in synchronize'"}, {"thread_id"=>40, "name"=>"[main]>worker10", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:insynchronize'"}, {"thread_id"=>43, "name"=>"[main]>worker13", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in synchronize'"}]}} [2017-05-09T14:10:19,179][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>321, "stalling_thread_info"=>{"other"=>[{"thread_id"=>35, "name"=>"[main]>worker5", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/awesome_print-1.7.0/lib/awesome_print/inspector.rb:156:inprintable'"}, {"thread_id"=>38, "name"=>"[main]>worker8", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in synchronize'"}, {"thread_id"=>40, "name"=>"[main]>worker10", "current_call"=>"[...]/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:insynchronize'"}]}}
I am not sure what to change in the logstash.yml to prevent this from happening the box its on has 16 cpus and 96GB of ram so I don't think its the resources.