# How :sql\_last\_value works with jdbc\_page\_size?

**URL:** <https://discuss.elastic.co/t/how-sql-last-value-works-with-jdbc-page-size/175861>\
**Category:** Logstash\
**Created:** [April 8, 2019, 2:52pm UTC](https://discuss.elastic.co/t/how-sql-last-value-works-with-jdbc-page-size/175861 "2019-04-08T14:52:02Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitaliikobrin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaliikobrin/32/43676_2.png) [@vitaliikobrin](https://discuss.elastic.co/u/vitaliikobrin)\
**Post date:** [April 8, 2019, 2:52pm UTC](https://discuss.elastic.co/t/how-sql-last-value-works-with-jdbc-page-size/175861/1 "2019-04-08T14:52:02Z")

</div>

I created logstash config to index data from MySQL DB in Elasticsearch by schedule.  
Here is my config:  
\> input {

> ```
> jdbc {
> jdbc_connection_string => "jdbc:mysql://mysql:3306/smartlikes_staging"
> jdbc_user => "root"
> jdbc_password => ""
> jdbc_driver_library => "jdbc-driver.jar"
> jdbc_driver_class => "com.mysql.jdbc.Driver"
> jdbc_page_size => 100000
> jdbc_paging_enabled => true
> use_column_value => true
> tracking_column => "updatedat"
> tracking_column_type => "timestamp"
> schedule => "*/2 * * * *"
> last_run_metadata_path => "/configs/products/.logstash_jdbc_last_run"
> statement => "SELECT id, title, brand, description FROM products WHERE updatedAt > :sql_last_value ORDER BY id"
> }
> }
> filter {
> mutate {
> add_field => { "[@metadata][product_id]" => "%{id}" }
> }
> mutate {
> remove_field => ["id"]
> }
> }
> output {
> stdout { codec => json_lines }
> elasticsearch {
> "hosts" => "elasticsearch:9200"
> "index" => "products"
> "doc_as_upsert" => true
> "action" => "update"
> "document_type" => "_doc"
> "document_id" => "%{[@metadata][product_id]}"
> }
> }
> 
> ```

The problem is:  
When I run Logstash I expect that it will load all items from the DB match the query (including pagination) and will send them page after page to the ES. And it will happen every 2 minutes. But actually what I have is: it runs approximately each 5 minutes and sends only one page to the ES at once. So, if I start Logstash and there is " **--- 2019-04-03 00:00:00.000000000 Z**" in _.logstash\_jdbc\_last\_run_ file it loads first 100.000 items and sends them to the ES. The value in _.logstash\_jdbc\_last\_run_ has not changed. Then it starts again after 5 minutes and sends next 100.000 items to the ES.  
When it finished, the value in _.logstash\_jdbc\_last\_run_ file has been successfully updated to the latest actual value from updatedAt column (that is ok).

It's not critically though. At least I can increase page size and make it run often. But seems something is going wrong here or my config is invalid.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 6, 2019, 2:52pm UTC](https://discuss.elastic.co/t/how-sql-last-value-works-with-jdbc-page-size/175861/2 "2019-05-06T14:52:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.