# Parallel execution for jdbc\_page\_size

**URL:** <https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260>\
**Category:** Logstash\
**Created:** [August 29, 2019, 5:03am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260 "2019-08-29T05:03:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrjayarajj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrjayarajj/32/51188_2.png) [@mrjayarajj](https://discuss.elastic.co/u/mrjayarajj)\
**Post date:** [August 29, 2019, 5:03am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260/1 "2019-08-29T05:03:16Z")

</div>

is it possible to run the jdbc\_page\_size execution in parallel ?

say if the count is 6K and jdbc\_page\_size is 1K, can we configure log stash to execute 6 parallel pagination ?

also if it get 100K , can we have max parallel execution , say like 10

actually my ask is without configuring multiple jdbc input plugin ?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 29, 2019, 8:57am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260/2 "2019-08-29T08:57:46Z")

</div>

No, this can't be done.

One reason for this is because we track the sql\_last\_value - this is a value taken from the results. For each row read it is updated and so its eventual value will be the value of the tracking column for the last row.

The jdbc input has a few different modes of operation and each one is selected via a combination of various settings, we don't do a good job at separating these into distinct code paths, they are quite intermingled. This make adding a parallel execution feature very difficult without refactoring the code completely.

It is of little help right now but we have plans to move the code to be a Java plugin that will share a lot of common code with the jdbc\_streaming and jdbc\_static filters. No ETA on when though.

---

<div class="post-metadata">

**Author:** ![mrjayarajj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrjayarajj/32/51188_2.png) [@mrjayarajj](https://discuss.elastic.co/u/mrjayarajj)\
**Post date:** [August 29, 2019, 8:36pm UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260/3 "2019-08-29T20:36:19Z")

</div>

Thanks for your comments, I tried to solve in other way like below

1. instead of jdbc\_page\_size, i used only jdbc\_fetch\_size
2. instead of one jdbc input plugin , i added ten , each sql execution based on the last digit of my primary key RIGHT(ST\_ID,1)=0 , RIGHT(ST\_ID,1)=1 , RIGHT(ST\_ID,1)=2 , etc
3. i maintain 10 different sql\_last\_value

this approch works for me, do you see any impact , since i am new to logstash.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [September 2, 2019, 8:30am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260/4 "2019-09-02T08:30:48Z")

</div>

Your solution is fine. I've seen many uses solve it in a similar fashion.

If the PK is numeric and alway auto-generated then a modulus function would also work for any number of jdbc input plugins.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2019, 8:31am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260/5 "2019-09-30T08:31:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.