I've been an idiot. I added a conditional into my Logstash pipelines which, long story short, has caused me to miss around an hour of logging data over about 18 different agents.
I've since fixed the issue and data is now being indexed correctly. I've read that if I delete the filebeat registry this will cause Filebeat to re-send the log files.
I have two concerns with this:
- Will this "duplicate" any events in Elasticsearchy that were indexed correctly?
- The if statement only broke certain modules so some log types were still indexed correctly. If I delete the registry would those files then be duplicated?
Duplicated data is probably even worse than the missing data so that is not an option. Even though this was only an hour we're probably talking 100,000+ events missing.
What is the right approach here to "backfill" this data?