I'm currently using the Elasticsearch input plugin to reindex an index via a Logstash (7.8.0) pipeline however I keep getting a random error which will stop the Logstash instance (and then get restarted, starting from the beginning again). The error is:
warning: thread "d1d9bd67d2f6b67650e4d28731599ca43bf2c429832ef20e110e54f795fc1185_slice_2" terminated with exception (report_on_exception is true):
Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:254
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131
scroll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/scroll.rb:61
scroll_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:317
process_next_scroll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:285
do_run_slice at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:277
do_run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:257
java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
There are multiple of these Logstash instances but only one seems to get hit with this error (but seems more susceptible to it from then on? weird). Is there a way I could increase how long it attempts to get a new connection from the pool?
Would increasing the scroll option help? (It's currently on 5m per batch of 10000 docs which seems ample as it should be indexing at 5000-10000 doc/s).
Furthermore none of my instances are currently cpu limited so I don't think it is related to pushing them too hard or anything like that. Any ideas would be appreciated.
Edit: It looks like they were just running out of memory!