# Logstash cannot handle huge amount of data **URL:** \ **Category:** Logstash\ **Created:** [January 4, 2019, 7:48am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906 "2019-01-04T07:48:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![longnx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longnx/32/39435_2.png) [@longnx](https://discuss.elastic.co/u/longnx)\ **Post date:** [January 4, 2019, 7:48am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/1 "2019-01-04T07:48:49Z")
Hi Experts, We are using Logstash to get data from Oracle database using JDBC with SQL query base on ID and time stamp. The Oracle table has around 15 billions/ month and the data inserted continuous to that table. It is around 1 millions new record / minute. Logstash cannot catch up the number of data that inserted to table. Can you suggest any solution to solve this issue? ---
**Author:** ![rijinmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rijinmp/32/24634_2.png) [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Post date:** [January 4, 2019, 10:01am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/2 "2019-01-04T10:01:57Z")
How much ram you are currently allocated to Logstash in jvm options ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 4, 2019, 10:05am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/3 "2019-01-04T10:05:12Z")
How complex is the query? How long does it take to run the query and retrieve the results if you use e.g. a script? If running the query and extracting the data is not the bottleneck, would it be possible to partition the query and have multiple pipelines each process a subset of the data and that way increase parallelism? ---
**Author:** ![longnx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longnx/32/39435_2.png) [@longnx](https://discuss.elastic.co/u/longnx)\ **Post date:** [January 7, 2019, 4:29am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/4 "2019-01-07T04:29:52Z")
> [@Christian\_Dahlqvist](#): > > query and extracting the data is not the bottleneck, would it be possible to partition the query and have multiple pipelines each process a subset of the data and that way increase parallelism? Hi Christian\_Dahlqvist The query join 3 tables and insert it to elasticsearch. The oracle database already created partition and indexes. How can I do multiple pipelines? ---
**Author:** ![longnx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longnx/32/39435_2.png) [@longnx](https://discuss.elastic.co/u/longnx)\ **Post date:** [January 7, 2019, 7:49am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/5 "2019-01-07T07:49:02Z")
I increase the RAM for it but nothing change ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 7, 2019, 8:02am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/6 "2019-01-07T08:02:00Z")
If you have a natural way to partition your data, you can create multiple pipelines where each has a jdbc input that contains a WHERE clause that ensures the inputs read different data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 4, 2019, 8:02am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906/7 "2019-02-04T08:02:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.