# When does Logstash update last\_run\_metadata\_path?

**URL:** <https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833>\
**Category:** Logstash\
**Created:** [June 14, 2021, 12:52pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833 "2021-06-14T12:52:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris\_ts24](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_ts24/32/79048_2.png) [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\
**Post date:** [June 14, 2021, 12:52pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833/1 "2021-06-14T12:52:27Z")

</div>

Hi all,

I am working with Logstash 7.13.0 & ES 7.13.0 to populate and maintain an index with records from an Oracle DB. The maintenance is achieved by running an update Logstash service every minute. The condition to be updated / inserted in the index is that the timestamp of the DB record is \>= timestamp last saved by logstash in the fille `last_run_metadata_path`.

The service is running without error, so now I'd like to ensure that I lose no records during service due to, for example, a server outage.

I see two methods to accomplish this:

1. [Persistent Queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#durability-persistent-queues), which are unfortunately inefficient due to disk writes.
2. When the server is back online, restart my Logstash service after refreshing the timestamp in`last_run_metadata_path` to a time **before** the server outage. Then all records during / after the outage will be updated / created in the index.

I see option 2 as the more efficient as server issues should be seldom occurring. According to the [Logstash Docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html): \* Upon query execution, this file ( `last_run_metadata_path`) will be updated with the current value of `sql_last_value`\*. My question is: when exactly is this value updated? Is it after records have been stored in the in-memory queue? Or after all records in the in-memory queue have been successfully inserted in the index?

For reference, the input section of my logstash configuration:

```auto
input {
		jdbc {
				jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
				jdbc_driver_library => "path\to\ojdbc8.jar"
				jdbc_connection_string => "jdbc:oracle:thin:@<DB_IP>:<DB_PORT>:<DB_SERVICE>"
		    jdbc_user => "<USER>"
		    jdbc_password => "<PASS>"
        statement_filepath => "path\to\update.sql"
				schedule => "* * * * *"
				last_run_metadata_path => "path\to\.logstash_jdbc_update"
     }
}

```

Thank you for your assistance!

Chris

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2021, 1:16pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833/2 "2021-06-14T13:16:51Z")

</div>

sql\_last\_value is [updated](https://github.com/logstash-plugins/logstash-input-jdbc/blob/878a73eb67b68366fac74fcc65e9cadd6cf44766/lib/logstash/plugin_mixins/jdbc/jdbc.rb#L255) after the query has been executed. The input knows nothing about whether the data has been indexed.

---

<div class="post-metadata">

**Author:** ![chris\_ts24](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_ts24/32/79048_2.png) [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\
**Post date:** [June 14, 2021, 1:21pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833/3 "2021-06-14T13:21:07Z")

</div>

Thank you @Badger. So that means that in the event of a server outage, the value must be updated to before the outage, as some records in the cycle that was interrupted may not have landed in the index.

Thanks again for the quick assist.

Chris

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2021, 1:21pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833/4 "2021-07-12T13:21:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.