"Timeout waiting for connection from pool" from http_poller plugin

[fr3eze]

I have a single logstash configuration with single http_poller plugin, and a lot of URL stacked inside:

input {
  http_poller {
    urls => {
      A => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      B => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      C => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      .....
    request_timeout => 1800
    socket_timeout => 1800
    schedule => { cron => "0 * * * * UTC"}
    codec => "json"
    tags => application_api
  }
}

filter {
  json {
    source => "message"
    add_field => {
        "Client" => "%{[devDetails][hdrTitle]}"
    }
    remove_field => [ "%{[devDetails][Type]}" ]
  }
  split { field => "[result]"}
  bytes {
    source => "[result][volume]"
    target => "bytes_numeric_field"
  }
  mutate {
    add_field => {
      "Application" => "%{[result][Name]}"
      "ApplicationVolume" => "%{[bytes_numeric_field]}"
    }
  }
  mutate {
    convert => {"ApplicationVolume" => "integer"}
    remove_field => [
      "[result]"
    ]
  }
}

output {
  elasticsearch {
    hosts => ["vpc-xxx:80"]
    manage_template => false
    index => "hourly_-%{+YYYY.MM}"
  }
}

As you can see I have A B C stacked under the urls block. I tested it was working fine with less than 10 urls defined. Nowadays i just increased it to more than 100, and I start to see error logs like:

[2020-04-21T17:48:28,674][DEBUG][logstash.pipeline ] filter received {"event"=>{"@version"=>"1", 
"http_request_failure"=>{"request"=>{"url"=>"https://xxx.xxx", "method"=>"get", "headers"=>
{"Accept"=>"application/json"}}, "error"=>"Timeout waiting for connection from pool", 
"name"=>"td_xxx_xxx", "runtime_seconds"=>316.753646, "backtrace"=>nil}, 
"@timestamp"=>2020-04-18T15:12:10.086Z, "tags"=>["_http_request_failure"]}}

And

[2020-04-21T17:52:43,187][DEBUG][logstash.pipeline        ] filter received {"event"=>
{"@version"=>"1", "http_request_failure"=>{"error"=>"SSL peer shut down incorrectly", 
"runtime_seconds"=>382.350346, "name"=>"scc", "request"=>{"url"=>"https://xxx.xxx", "headers"=>
{"Accept"=>"application/json"}, "method"=>"get"}, "backtrace"=>nil}, "tags"=>
["_http_request_failure"], "@timestamp"=>2020-04-18T20:06:40.568Z}}

I've increased the request_timeout and socket_timeout to 1800 but seems like it does ot help much.

[Badger]

@fr3eze

i just increased it to more than 100

The pool_max option defaults to 50, so if you are trying to poll 100 URIs then only 50 of them can be running at the same time. You can see your requests were running for over 300 seconds, which suggests some of your URIs are slow to respond.

[fr3eze]

I missed that setting, thanks for the reminder. Now I've set the pool_max to 250 while keeping the request_timeout and socket_timeout at 1800, still see the Timeout waiting for connection from pool error. What am I missing here?

The runtime is accumulative over the stacked URLs as they are executed one by one in serial manner, no?

[Badger]

I do not know the answer to either question.

[fr3eze]

Issue resolved.

It turns out my Logstash server was really tight on resources, especially memory. Assigned more memory to the server and tune the performance according to https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#profiling-the-heap. Above mentioned errors all gone.