Hi. New to elasticsearch but my cluster (with 1 node) has around 3,000 shards already. Upon reading, I wanted to snapshot my previous monthly indices from logstash and delete them, thus making space for shards.
I've read that snapshots are incremental but will keep data if it references other data.
Problem is, we are still on the process of 'backfilling' our logs. Some (or most) of our logs are still not inside elasticsearch. Say 50% of November 2016 is there and the other 50% of the logs are not there. I'm kinda limited with space especially with the shard problems so I wanted to 'snapshot-delete-backfill' so I can keep my node/cluster green.
I want to snapshot a whole month of data. The problem is that not all data are in elasticsearch when I do the snapshot. I also cannot fill it up to 100% since it would double the shard count. So I want to snapshot it at 50%, delete the snapshotted indices and backfill again. Rinse and repeat.
My question is, is this possible or an efficient solution to my problem?
Using ELK 5.1.