# Logstash skip records by adding latest updated record date in last run metadata file

**URL:** <https://discuss.elastic.co/t/logstash-skip-records-by-adding-latest-updated-record-date-in-last-run-metadata-file/209309>\
**Category:** Logstash\
**Created:** [November 25, 2019, 1:23pm UTC](https://discuss.elastic.co/t/logstash-skip-records-by-adding-latest-updated-record-date-in-last-run-metadata-file/209309 "2019-11-25T13:23:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![saifrehman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saifrehman/32/56971_2.png) [@saifrehman](https://discuss.elastic.co/u/saifrehman)\
**Post date:** [November 25, 2019, 1:23pm UTC](https://discuss.elastic.co/t/logstash-skip-records-by-adding-latest-updated-record-date-in-last-run-metadata-file/209309/1 "2019-11-25T13:23:23Z")

</div>

My goal is to set up continuous data migration from MySQL to Elasticsearch. Everything is working fine but I've stuck with :sql\_last\_value. My server and database timezone is in UTC format.

Problem: When i update 4 to 6 records at mysql side. logstash should update those records in Elasticsearch. But logstash skipping two or three records. but putting the lastest updated record date in .logstash\_jdbc\_last\_run file.

Example: I have update 4 records from mysql side. These records update\_at date is as follow

> ```
> 2019-11-25 12:29:01
> 2019-11-25 12:29:58
> 2019-11-25 12:29:59
> 2019-11-25 12:29:00
> 
> ```

logstash update the date in logstash\_jdbc\_last\_run file which is '2019-11-25 12:29:01' and skip the 2 records which has dates as follow

> ```
> 2019-11-25 12:29:59
> 2019-11-25 12:29:00
> 
> ```

Logstash logs it in local timezone, so next time it queries db, :sql\_last\_value value is correct.

So if the last received datetime field from db was like 2019-11-25 12:29:01, next query to db will be for :sql\_last\_value \> 2019-11-25 12:29:01 and it missed the records with a timeframe of 3 to 5 seconds.

---

<div class="post-metadata">

**Author:** ![saifrehman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saifrehman/32/56971_2.png) [@saifrehman](https://discuss.elastic.co/u/saifrehman)\
**Post date:** [November 25, 2019, 1:28pm UTC](https://discuss.elastic.co/t/logstash-skip-records-by-adding-latest-updated-record-date-in-last-run-metadata-file/209309/2 "2019-11-25T13:28:01Z")

</div>

Following is my config file  
@magnusbaeck please reply

> input{  
> jdbc {  
> jdbc\_connection\_string =\> "----"  
> jdbc\_user =\> "------"  
> jdbc\_password =\> "-----------"  
> jdbc\_validate\_connection =\> true  
> #jdbc\_driver\_library =\> "/home/ubuntu/mysql-connector-java-5.1.36-bin.jar"  
> jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"  
> tracking\_column =\> "updated\_timestamp"  
> tracking\_column\_type =\> "timestamp"  
> use\_column\_value =\> true  
> jdbc\_default\_timezone =\> "UTC"  
> schedule =\> "\* \* \* \* \* \*"  
> statement =\> "SELECT  
> id as db\_id,  
> DATE\_FORMAT(updated\_timestamp, '%Y-%m-%d %T') as updated\_timestamp,  
> CONCAT('u','-', following\_id) as user\_id,  
> created\_at,   
> CONCAT('fg','-', id) as id,  
> CONCAT('u','-', follower\_id) as object\_id,  
> id,  
> status  
> FROM  
> friends  
> WHERE  
> updated\_timestamp \> :sql\_last\_value order by updated\_timestamp DESC "   
> last\_run\_metadata\_path =\> "/home/ubuntu/logstash\_track\_date/.logstash\_followings\_jdbc\_last\_run"  
> add\_field =\> { "doctype" =\> "followings" }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2019, 1:28pm UTC](https://discuss.elastic.co/t/logstash-skip-records-by-adding-latest-updated-record-date-in-last-run-metadata-file/209309/3 "2019-12-23T13:28:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.