I'm a little confused on the behavior of the snapshotting process, and how to configure it to support our use case.
In our use case, we are collecting time-based log events from a variety of different servers that we host, and we need to retain this log data permanently in order to report (and possibly reproduce) usage statistics on our website. The log data for each server is indexed into a separate index, and each index is rolled-over when it reaches a specific size or age limit. Thus, we have a growing list of indices that looks as follows:
- load-balancer-A-000001, -000002, etc.
- load-balancer-B-000001, -000002, etc.
- apache-A-000001, -000002, etc.
- ...and many, many more...
From my understanding of the documentation (and other discussions here), each incremental snapshot only includes references to the indices that changed since the last snapshot. Further, we must specify the "Number of snapshots to retain," for which the upper limit appears to be 200.
Given that all of our indices have already rolled-over at least once, if we were to begin taking daily snapshots starting tomorrow, does this mean that after 201 days we will lose all of the data for indices prior to their first rollover (e.g., "load-balancer-A-000001"), since they didn't change between the first and second snapshots? If so, how do we get around this limitation?