Cross posting this from a r/kibana post I made that has yet to receive any responses
Maybe a bit of an oddball question here. I have a workflow that automatically downloads data from a federal website each week, processes it a bit, and posts it on a Kibana dashboard. The dashboard contains all the weekly files going back as long as I've been collecting them. The data file is mostly the same each week, but with changes for a small percentage of records that are important to track (and which records see changes is different each week) and I retain all of the records in each weekly file for every week so the status of each record can always be known for a given week. I have a counter on the dashboard that counts the number of records meeting a certain condition in each weekly file, and the way I make the counter only include the most recent data is to set a condition that only includes data that was posted in the last calendar week.
The challenge is that the federal agency seems to post the file manually each week and doesn't always do it on the same day and sometimes has weeks where they do not post any file for some reason or another. When it has been more than a week since the last data file was posted, my counter then drops to zero, and I haven't been able to figure out a way to get the counter to only include data from the most recent file, regardless of how long it has been since that file has been posted. Any ideas of how to do this?