I am pulling entities out of a database and serializing them to .json using a C# console app.
The .json files are then read by Filebeat which sends the data to LogStash, which pushes into Elasticsearch.
Every 5 minutes I will query the database for entities that have changed.
Once a day (or week) I would like to completely rebuild the entire index, just to make sure we haven't missed anything.
I understand the concept of aliases and intend to use one for this.
How should I configure Logstash to push simple changes to the existing index, but push the complete rebuild to the next version?
I thought about putting the day (or week-depending on the rebuild threshold) timestamp in the index name of the Logstash Filebeat-to-Elasticsearch config, but this would simply create a new version, I wouldn't know when to swap out old for new.
I could have my console app handle this, but I'm not sure how it would know that Logstash was done pushing files to the new index...
Also, it would be nice and maybe even imperative to be able to queue up an entire rebuild on-demand. I am new to Elasticsearch and I fear I may need to be react quickly in the event I do something wrong.
Looking for some seasoned input. Maybe I'm thinking about this all wrong and there is a better route to go.