# "Timeout waiting for connection from pool" from http\_poller plugin

**URL:** <https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014>\
**Category:** Logstash\
**Created:** [April 21, 2020, 10:32am UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014 "2020-04-21T10:32:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fr3eze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fr3eze/32/63600_2.png) [@fr3eze](https://discuss.elastic.co/u/fr3eze)\
**Post date:** [April 21, 2020, 10:32am UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/1 "2020-04-21T10:32:26Z")

</div>

I have a single logstash configuration with single http\_poller plugin, and a lot of URL stacked inside:

```auto
input {
  http_poller {
    urls => {
      A => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      B => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      C => {
        method => get
        url => "https://xxx.xxx"
        headers => {
          Accept => "application/json"
        }
      }
      .....
    request_timeout => 1800
    socket_timeout => 1800
    schedule => { cron => "0 * * * * UTC"}
    codec => "json"
    tags => application_api
  }
}

filter {
  json {
    source => "message"
    add_field => {
        "Client" => "%{[devDetails][hdrTitle]}"
    }
    remove_field => ["%{[devDetails][Type]}" ]
  }
  split { field => "[result]"}
  bytes {
    source => "[result][volume]"
    target => "bytes_numeric_field"
  }
  mutate {
    add_field => {
      "Application" => "%{[result][Name]}"
      "ApplicationVolume" => "%{[bytes_numeric_field]}"
    }
  }
  mutate {
    convert => {"ApplicationVolume" => "integer"}
    remove_field => [
      "[result]"
    ]
  }
}

output {
  elasticsearch {
    hosts => ["vpc-xxx:80"]
    manage_template => false
    index => "hourly_-%{+YYYY.MM}"
  }
}

```

As you can see I have `A` `B` `C` stacked under the `urls` block. I tested it was working fine with less than 10 urls defined. Nowadays i just increased it to more than 100, and I start to see error logs like:

```auto
[2020-04-21T17:48:28,674][DEBUG][logstash.pipeline] filter received {"event"=>{"@version"=>"1", 
"http_request_failure"=>{"request"=>{"url"=>"https://xxx.xxx", "method"=>"get", "headers"=>
{"Accept"=>"application/json"}}, "error"=>"Timeout waiting for connection from pool", 
"name"=>"td_xxx_xxx", "runtime_seconds"=>316.753646, "backtrace"=>nil}, 
"@timestamp"=>2020-04-18T15:12:10.086Z, "tags"=>["_http_request_failure"]}}

```

And

```auto
[2020-04-21T17:52:43,187][DEBUG][logstash.pipeline] filter received {"event"=>
{"@version"=>"1", "http_request_failure"=>{"error"=>"SSL peer shut down incorrectly", 
"runtime_seconds"=>382.350346, "name"=>"scc", "request"=>{"url"=>"https://xxx.xxx", "headers"=>
{"Accept"=>"application/json"}, "method"=>"get"}, "backtrace"=>nil}, "tags"=>
["_http_request_failure"], "@timestamp"=>2020-04-18T20:06:40.568Z}}

```

I've increased the `request_timeout` and `socket_timeout` to 1800 but seems like it does ot help much.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 21, 2020, 3:24pm UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/2 "2020-04-21T15:24:10Z")

</div>

> [@fr3eze](#):
>
> i just increased it to more than 100

The pool\_max option defaults to 50, so if you are trying to poll 100 URIs then only 50 of them can be running at the same time. You can see your requests were running for over 300 seconds, which suggests some of your URIs are slow to respond.

---

<div class="post-metadata">

**Author:** ![fr3eze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fr3eze/32/63600_2.png) [@fr3eze](https://discuss.elastic.co/u/fr3eze)\
**Post date:** [April 22, 2020, 1:20am UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/3 "2020-04-22T01:20:57Z")

</div>

I missed that setting, thanks for the reminder. Now I've set the `pool_max` to 250 while keeping the `request_timeout` and `socket_timeout` at 1800, still see the `Timeout waiting for connection from pool` error. What am I missing here?

The runtime is accumulative over the stacked URLs as they are executed one by one in serial manner, no?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 22, 2020, 3:41pm UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/4 "2020-04-22T15:41:41Z")

</div>

I do not know the answer to either question.

---

<div class="post-metadata">

**Author:** ![fr3eze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fr3eze/32/63600_2.png) [@fr3eze](https://discuss.elastic.co/u/fr3eze)\
**Post date:** [April 24, 2020, 1:50am UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/5 "2020-04-24T01:50:57Z")

</div>

Issue resolved.

It turns out my Logstash server was really tight on resources, especially memory. Assigned more memory to the server and tune the performance according to [https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#profiling-the-heap](https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#profiling-the-heap). Above mentioned errors all gone.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2020, 2:01am UTC](https://discuss.elastic.co/t/timeout-waiting-for-connection-from-pool-from-http-poller-plugin/229014/6 "2020-05-22T02:01:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.