Elasticsearch daily index, action update, jdbc input, data getting updated on the next day

Hi, I recently started working with elasticsearch.

So, I have a logstash config with jdbc as the input trying to fetch the data from a table. (select statement of some specific fields where lastupdatedon >= :sql_last_value) for every 30 minutes

createdon and lastupdatedon are date type columns.

and the output is to elasticsearch to create the daily index and action is update with a tracking column "lastupdatedon"

the table data is like this

As soon as the data is entered into the table, the status column value would be as "queued". then later sometimes on the same day or sometimes the next day it gets updated to "cont" or "notcont".

All the data will be updated at somepoint of the same day or the day after.

So, If I see in kibana I am getting the docs, but I see a lot of them as queued status. I found out that since I am creating a new index daily, the data that is getting updated the next day is not getting updated in the documents.

Please let me know my options and what can be done to avoid this.

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.