# JDBC\_Streaming Filter Plugin PoolTimeout Issue **URL:** \ **Category:** Logstash\ **Created:** [January 12, 2018, 10:31pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380 "2018-01-12T22:31:37Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![msugas19](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/msugas19/32/26627_2.png) [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Post date:** [January 12, 2018, 10:31pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/1 "2018-01-12T22:31:37Z")
Hello everyone. I am pulling data into logstash using the JDBC input plugin filter with a fetch size of 10,000 and for each of those records I am trying to perform another query to get child attributes for the parent document. The problem I am facing is that I keep receiving the following error from logstash: [timestamp] [WARN] [logstash.filters.jdbcstreaming] Exception when executing JDBC query {:exception=\>#\} Most of my documents are being tagged as having jdbc\_streaming errors and they don't end up executing the queries as far as I can tell. How can I remedy this? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 15, 2018, 4:48pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/2 "2018-01-15T16:48:49Z")
From an issue on the Sequel repo... > PoolTimeout is raised when Sequel needs a connection from the connection pool, but the pool already has the maximum number of created connections, and all of them are already checked out by other threads. > In any case, this doesn't appear to be a bug in Sequel, as Sequel is correctly raising the PoolTimeout exception after the configured amount of time has passed. The pool size is set by the `max_connections` setting, defaults to 4. In the initial released version of the `jdbc_streaming` plugin this cannot be adjusted. You can try to reduce the number of LS workers to 4 though. **How long do your lookup queries take to execute?** These lookup queries are supposed to be fast single table, single record queries with the WHERE clause using an indexed column. ---
**Author:** ![msugas19](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/msugas19/32/26627_2.png) [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Post date:** [January 16, 2018, 5:35pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/3 "2018-01-16T17:35:34Z")
Will try out reducing the amount of worker threads. Are there any plans in the future to allow more connections in this filter? We have a complex data use case and using the streaming as a full-fledged query service could help solve them ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 16, 2018, 5:40pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/4 "2018-01-16T17:40:33Z")
Well, yes, I will add the settings to increase the pool size over the next few weeks. Can you improve the statement performance? Materialized View? Copy the table(s) to a simplified structure/record count with indexing. **What is the cardinality of the lookup identifier?** Ideally you want low cardinality for useful caching. Bear in mind that this is the scenario: Threads 1-4 uses Pool.1-4 to make lookup queries. Thread 5-8 waits to get access to one of Pool.1-4. 5 seconds later... Thread 5 gets a timeout exception. Thread 1 returns Pool.1 to the pool. Threads 6 get Pool.1 and makes the lookup query. ---
**Author:** ![msugas19](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/msugas19/32/26627_2.png) [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Post date:** [January 16, 2018, 5:55pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/5 "2018-01-16T17:55:30Z")
Potentially in the future we can do these things, however for our use case right now we cannot as this is a POC project and my team does not own the tables we are querying from. The inner JDBC query is a join across three tables that generally takes less than 10s to return ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 16, 2018, 6:48pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/6 "2018-01-16T18:48:50Z")
That is way too long. Should be milliseconds! Are the 3 tables static, or frequently updated (\> once a day)? Another option is to have 3 jdbc\_streaming filters one for each table - with a field value added by the first jdbc\_streaming filter used in the second jdbc\_streaming filter etc. I have not tried 3 jdbc\_streaming filters running in the same LS instance though. If you have success with 3 filters, please let me know. Unfortunately I don't have the DB resources and data sets to test this for myself. Cardinality? ---
**Author:** ![msugas19](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/msugas19/32/26627_2.png) [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Post date:** [January 16, 2018, 8:03pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/7 "2018-01-16T20:03:10Z")
If you think that is long, you need to see how slow Oracle can get 😫 The problem with the 3 streaming approach is we need the distinct combination of fields across three different tables. I am sure we could get this done client-side, however the reason we are using logstash for ingestion of this magnitude in the first place is to try and avoid creating programmatic solutions as we do this in other sectors and it is very costly. The cardinality of the lookup query can be anywhere from 1:1 to 30:1. It varies on the input. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 16, 2018, 8:38pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/8 "2018-01-16T20:38:33Z")
I get it. One other design: LS1: JDBC input (collect the three table join) -\> ES (index: static3). LS 2: JDBC input (original query) -\> ES filter (index query on static3) -\> ES (index: original index) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 13, 2018, 8:39pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380/9 "2018-02-13T20:39:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.