# Logstash reading sql server data real time **URL:** \ **Category:** Logstash\ **Created:** [February 6, 2019, 9:21pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357 "2019-02-06T21:21:30Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Pradyumna\_Jha](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Post date:** [February 6, 2019, 9:21pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/1 "2019-02-06T21:21:30Z")
Is there any way i can configure logstash so that it picks up delta records real time automatically. If not then is there any opensource plugin/tool available to achieve this? I am pushing log data stored in sql DB to elasticsearch but logstash doesnot keep on running to push new records to elasticsearch. I dont want to run it again or to schedule it. Is there any configuration so that it keeps listening for new records as it does for files. In need of quick response please. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 6, 2019, 10:04pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/2 "2019-02-06T22:04:36Z")
What issue do you have using a jdbc input with a [schedule](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_scheduling_2)? ---
**Author:** ![Pradyumna\_Jha](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Post date:** [February 7, 2019, 5:19pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/3 "2019-02-07T17:19:34Z")
There is no harm but just thinking if it would affect or put a load on msql server. Also, if i schedule it, how would I be able to make sure that i am picking up only new records in subsequent runs keeping in mind that I dont have any primary key in the table and also no combination of columns would make it unique too. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2019, 7:10pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/4 "2019-02-07T19:10:15Z")
> [@Pradyumna\_Jha](#): > > how would I be able to make sure that i am picking up only new records in subsequent runs The jdbc input can maintain [state](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_state). It requires that you have a either a timestamp column, or a sequence-like column so that you can use a query where you fetch all records newer than the last time it ran, or all records with sequence number higher than the one used the last time it ran. ---
**Author:** ![Pradyumna\_Jha](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Post date:** [February 7, 2019, 7:44pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/5 "2019-02-07T19:44:07Z")
So I have a timestamp column. But the only fear is that there is going to be a lots of data coming in and i will be scheduling logstash to every 5 secs. Do u think that this might create a overlapping of data or missing some data issue. If not then it would be really helpful if provide the conf file input configuration or any link which has this example. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2019, 9:07pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/6 "2019-02-07T21:07:23Z")
The [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html) has a couple of examples. With any scheme that remembers the last timestamp there is going to be a possibility of missing data. If you have a sequence then I think that possibility disappears. ---
**Author:** ![Pradyumna\_Jha](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Post date:** [February 7, 2019, 9:59pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/7 "2019-02-07T21:59:34Z")
Do i need to have sequence column in my table or is there a way logstash can create and maintain it internally? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2019, 10:06pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/8 "2019-02-07T22:06:37Z")
logstash cannot do it, it would have to be in the db. ---
**Author:** ![Pradyumna\_Jha](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Post date:** [February 7, 2019, 10:08pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/9 "2019-02-07T22:08:05Z")
Hmm... Then with my requirement and answers from you I am back to my original question. Thanks for answering all my queries. It was really helpful and certainly saved my time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2019, 10:08pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357/10 "2019-03-07T22:08:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.