Hi, I recently started working with elasticsearch.
So, I have a logstash config with jdbc as the input trying to fetch the data from a table. (select statement of some specific fields where lastupdatedon >= :sql_last_value) for every 30 minutes
createdon and lastupdatedon are date type columns.
and the output is to elasticsearch to create the daily index and action is update with a tracking column "lastupdatedon"
the table data is like this
As soon as the data is entered into the table, the status column value would be as "queued". then later sometimes on the same day or sometimes the next day it gets updated to "cont" or "notcont".
All the data will be updated at somepoint of the same day or the day after.
So, If I see in kibana I am getting the docs, but I see a lot of them as queued status. I found out that since I am creating a new index daily, the data that is getting updated the next day is not getting updated in the documents.
Please let me know my options and what can be done to avoid this.
Thanks