# Logstash JDBC input plugin
**URL:** https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994
**Category:** Logstash
**Created:** [June 1, 2017, 10:10pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994 "2017-06-01T22:10:01Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@asharma](https://discuss.elastic.co/u/asharma)
#### Post date: [June 1, 2017, 10:10pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994/1 "2017-06-01T22:10:02Z")
I am using the jdbc plugin to import data form AMAZON redshift to elasticsearch using logstash.
I am processing incremental updates for a very big table which adds around 2 million rows every hour and has a timestamp attached to each row.
I am facing a problem where since data from redshift is not coming in sorted order, in order to process batch update, using :sql\_last\_value i have to filter latest 2 million row and then sort it which is taking a lot of time.
Is there any work around for this problem so that the sql\_last\_value stores the max of the current processed batch rather than storing the last value which requires the input to be sorted on that column assigned to sql\_last\_value ?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 5:15am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994/2 "2017-06-02T05:15:50Z")
> I am facing a problem where since data from redshift is not coming in sorted order, in order to process batch update, using :sql\_last\_value i have to filter latest 2 million row and then sort it which is taking a lot of time.
Can't you let the jdbc input run more often than once an hour so that each batch becomes smaller?
> Is there any work around for this problem so that the sql\_last\_value stores the max of the current processed batch rather than storing the last value
Sorry, I don't understand the difference.
> which requires the input to be sorted on that column assigned to sql\_last\_value ?
If you're only using a timestamp from a column to keep track of what has been processed I don't see how you can possibly avoid sorting the rows before processing them.
---
### Author:  [@asharma](https://discuss.elastic.co/u/asharma)
#### Post date: [June 2, 2017, 3:16pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994/3 "2017-06-02T15:16:22Z")
For the second part, since the rows returned are not in sorted order, what value does :sql\_last\_value srore for the timestamp column assigned to it ? Will it be the timestamp of the last processed row (which might not be the latest time stamp because of redshift) or will it store the maximum of the timestamps processed in the current batch ??
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [June 2, 2017, 3:57pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994/4 "2017-06-02T15:57:45Z")
Its the timestamp of the last processed row. You need to sort it.
Does Magnus' suggestion of more frequent scheduling not work for you?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2017, 3:58pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin/87994/5 "2017-06-30T15:58:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.