# How to run queries from multiple JDBC inputs sequentially

**URL:** <https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226>\
**Category:** Logstash\
**Created:** [March 2, 2017, 10:04pm UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226 "2017-03-02T22:04:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![madebydna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madebydna/32/15035_2.png) [@madebydna](https://discuss.elastic.co/u/madebydna)\
**Post date:** [March 2, 2017, 10:04pm UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226/1 "2017-03-02T22:04:55Z")

</div>

My pipeline consists of three JDBC inputs, a set of common filters and one output stage to Elasticsearch. The three JDBC inputs fetch similar data from three different tables, the aggregate filter groups the data by a particular field that is present in all three tables, while the output filter dynamically sets the type based on which table the data originated from.

```
input {
   jdbc {
      statement => "Select foo FROM table_a ORDER BY foo;"
   }
  jdbc {
      statement => "Select foo FROM table_b ORDER BY foo;"
   }
  jdbc {
      statement => "Select foo FROM table_c ORDER BY foo;"
   }
}

filters {
   aggregate {
      task_id => "%{foo}"
      code => "...."
      push_previous_map_as_event => true
   }
}

output {
    elasticsearch {
       action => "index"
       document_id => "%{foo}"
       document_type => "%{[@metadata][type]}"
    }
}

```

My problem is that the 3 queries are not executed sequentially, but in parallel, or at least in quick succession. This has the effect that the results / events arrive interleaved, e.g. a few events from table\_a followed by events from table\_b, then back to table\_a, etc. This interferes with the workings of the aggregate filter which expects all events from table\_a ordered by foo, then all events from table\_b ordered by foo, etc.

I've been setting the worker count to 1 when running the pipeline, but upon closer reading of the documentation I realized that this setting only affects the filter and output stages of the pipeline.

I read that "Each input stage in the Logstash pipeline runs in its own thread." In my particular case, do I have other options besides running each input separately?

A [similar question](https://discuss.elastic.co/t/synchronize-multiple-jdbc-inputs-pool-session-error/68837) was asked before but didn't receive an answer.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 3, 2017, 4:35am UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226/2 "2017-03-03T04:35:36Z")

</div>

You'll want to tag each input, then use a agg filter with a conditional for each to make sure they are grouped.

---

<div class="post-metadata">

**Author:** ![Randy\_BoBandy](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@Randy\_BoBandy](https://discuss.elastic.co/u/Randy_BoBandy)\
**Post date:** [March 21, 2017, 2:17pm UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226/3 "2017-03-21T14:17:56Z")

</div>

Does this mean it is not possible to run separate inputs sequentially? I'm seeing indexing performance drop by a factor of 10 when the number of inputs increases.  
I've tried limiting the worker processes, but this seems to have no effect on query execution.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 21, 2017, 8:18pm UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226/4 "2017-03-21T20:18:35Z")

</div>

> [@Randy\_BoBandy](#):
>
> Does this mean it is not possible to run separate inputs sequentially?

Yes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 18, 2017, 8:18pm UTC](https://discuss.elastic.co/t/how-to-run-queries-from-multiple-jdbc-inputs-sequentially/77226/5 "2017-04-18T20:18:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.