# Logstash update sql\_last\_value while using paging

**URL:** <https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362>\
**Category:** Logstash\
**Created:** [June 19, 2023, 12:00pm UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362 "2023-06-19T12:00:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zalseryani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zalseryani/32/122438_2.png) [@zalseryani](https://discuss.elastic.co/u/zalseryani)\
**Post date:** [June 19, 2023, 12:00pm UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362/1 "2023-06-19T12:00:33Z")

</div>

# sql\_last\_value update with Paging

- I am configuring logstash to use jdbc input knowing that I am using jdbc\_paging with the configuration.

- what I am facing now is that sql\_last\_value is being updated after all records in the db is completed which is not what I want, I need to have sql\_last\_value being updated between paging such that in case logstash pod is down for any reason, it will be able to continue from the sql\_last\_value for which the last paging query finished successfully.

- Is this something possible?

- when I am not using the paging, and only default jdbc input, it is able to do that and I can see the last\_metadata\_path\_file is being updated (using watch cat )

# Current Logstash Configuration

```auto
        jdbc_user =>{{ include "logstash.database_username" . }}
        jdbc_password =>{{ include "logstash.database_password" . }}
        schedule => "*/5 * * * * *"
        statement => "select id,msg_text from msg_source where id > :sql_last_value"
        jdbc_paging_enabled => true
        jdbc_page_size => 2000
        use_column_value => true
        tracking_column_type => "numeric"
        tracking_column => "id"
        last_run_metadata_path => "/usr/share/logstash/pmw-sql-last-value/{{ .Release.Name }}/messages_sql_last_value.yml"

```

- I can see that many users are facing the same case.

I really appreciate any help you can provide.

---

<div class="post-metadata">

**Author:** ![zalseryani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zalseryani/32/122438_2.png) [@zalseryani](https://discuss.elastic.co/u/zalseryani)\
**Post date:** [June 19, 2023, 2:38pm UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362/2 "2023-06-19T14:38:19Z")

</div>

Update:

I have tested my configuration on MSSQL DB and It is updating correctly, but when I tested it on oracle db, with paging, the sql\_last\_value is not updated on the file used in last\_run\_metadata\_path until all no more records are available in DB.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2023, 4:22pm UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362/3 "2023-06-19T16:22:46Z")

</div>

> [@zalseryani](#):
>
> what I am facing now is that sql\_last\_value is being updated after all records in the db is completed which is not what I want

That is just how it works. The [paged statement handler](https://github.com/logstash-plugins/logstash-integration-jdbc/blob/c06e3192d8da3528bf4fc045b3fd8721b60a5a1e/lib/logstash/plugin_mixins/jdbc/statement_handler.rb#L110) loops over pages until it gets a page that is not full. The input [updates sql\_last\_value](https://github.com/logstash-plugins/logstash-integration-jdbc/blob/c06e3192d8da3528bf4fc045b3fd8721b60a5a1e/lib/logstash/plugin_mixins/jdbc/jdbc.rb#L235) after the statement handler completes the query.

---

<div class="post-metadata">

**Author:** ![zalseryani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zalseryani/32/122438_2.png) [@zalseryani](https://discuss.elastic.co/u/zalseryani)\
**Post date:** [June 20, 2023, 6:00am UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362/4 "2023-06-20T06:00:52Z")

</div>

@Badger

Thank you for your fast response.

But why it is working with MSSQL and not with Oracle DB? Could be that ojdbc needs some parameters to be passed?

I can see that file is being updated while the paging is processing 10K messages but with Oracle, it is not.

Much Respect.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2023, 6:01am UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362/5 "2023-07-18T06:01:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.