Is there any way i can configure logstash so that it picks up delta records real time automatically. If not then is there any opensource plugin/tool available to achieve this? I am pushing log data stored in sql DB to elasticsearch but logstash doesnot keep on running to push new records to elasticsearch. I dont want to run it again or to schedule it. Is there any configuration so that it keeps listening for new records as it does for files. In need of quick response please.
There is no harm but just thinking if it would affect or put a load on msql server.
Also, if i schedule it, how would I be able to make sure that i am picking up only new records in subsequent runs keeping in mind that I dont have any primary key in the table and also no combination of columns would make it unique too.
The jdbc input can maintain state. It requires that you have a either a timestamp column, or a sequence-like column so that you can use a query where you fetch all records newer than the last time it ran, or all records with sequence number higher than the one used the last time it ran.
So I have a timestamp column. But the only fear is that there is going to be a lots of data coming in and i will be scheduling logstash to every 5 secs. Do u think that this might create a overlapping of data or missing some data issue. If not then it would be really helpful if provide the conf file input configuration or any link which has this example.
The documentation has a couple of examples.
With any scheme that remembers the last timestamp there is going to be a possibility of missing data. If you have a sequence then I think that possibility disappears.
1 Like
Do i need to have sequence column in my table or is there a way logstash can create and maintain it internally?
logstash cannot do it, it would have to be in the db.
1 Like
Hmm... Then with my requirement and answers from you I am back to my original question. Thanks for answering all my queries. It was really helpful and certainly saved my time.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.