# SQL execute for full table

**URL:** <https://discuss.elastic.co/t/sql-execute-for-full-table/151708>\
**Category:** Logstash\
**Created:** [October 9, 2018, 8:45pm UTC](https://discuss.elastic.co/t/sql-execute-for-full-table/151708 "2018-10-09T20:45:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [October 9, 2018, 8:45pm UTC](https://discuss.elastic.co/t/sql-execute-for-full-table/151708/1 "2018-10-09T20:45:00Z")

</div>

I have configuration file that works pulling data from oracle database. document\_id also works and no more duplicate but everytime It runs it does full tablescan and takes forever to pull the data because table has millions of row. I understand if it runs long time first time. But shouldn't it run only short while when it pulls new record only

here is config file  
input {  
jdbc {  
jdbc\_validate\_connection =\> true  
jdbc\_connection\_string =\> "jdbc:oracle:thin:@/DBID"  
jdbc\_user =\> "USER"  
jdbc\_password =\> "Password"  
jdbc\_driver\_library =\> "/usr/lib/oracle/12.2/client64/lib/ojdbc8.jar"  
jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver"  
statement =\> "select JOB, STATUS from JOBS where extract(year from SUBMITTED) \> 2015 and JOB \> :sql\_last\_value"  
last\_run\_metadata\_path =\> "/tmp/logstash.lastrun"  
use\_column\_value=\>true  
tracking\_column=\>"job"  
record\_last\_run =\> true  
schedule =\> "20 \*/5 \* \* \*"  
}  
}

filter {  
}

output {  
elasticsearch {  
hosts =\> ["elktst01:9200"]  
index =\> "jobs\_all\_test1"  
document\_id =\> "%{job}"  
}  
stdout { codec =\> rubydebug }  
}

It pulls 64Million record first time and completed in few hour. but when it starts up second time it keeps running for hours and only adds few hundreds record which are expected.

Only problem is that it runs forever.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [October 10, 2018, 2:06pm UTC](https://discuss.elastic.co/t/sql-execute-for-full-table/151708/2 "2018-10-10T14:06:03Z")

</div>

Anyone is facing this problem?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [October 10, 2018, 5:39pm UTC](https://discuss.elastic.co/t/sql-execute-for-full-table/151708/3 "2018-10-10T17:39:33Z")

</div>

Well I jump the gun. it is working as it expacted. each query is completing in just seconds and getting me only new records.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 7, 2018, 5:44pm UTC](https://discuss.elastic.co/t/sql-execute-for-full-table/151708/4 "2018-11-07T17:44:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.