Continuing the discussion from ELK stack questions:
First I would like to say sorry for posting in the wrong category, I wasn't sure where to do it.
Let me answer your questions:
I would like to use the two primary nodes for searches in the logs for the last three months. The idea of the third one is to act as a backup if something goes wrong with the first two + for searches on indexes that are older than 3 months.
_source yes, but fields like _message - i am not sure. There will be also custom fields with longer text and I'm wondering if they are compressed the search will be slower than if they are not compressed.
It's obviously too much, because 30x30 means nearly 1 TB per month. This would result in ~12TB per year raw logs, and keeping in mind that we must hold logs for AT LEAST one year back, well, its obviously too much. At the moment, I am solving this with gzipping data older than 7 days, which does a good job. However, I am not sure how can I achieve such result in elasticsearch.
The idea with ES snapshots came up in my mind when I was searching a way to shrink some data older than X months. Let's say that a user wants to search for a data older than 3 months - I would like to route it to the replica shard and somehow give him the ability to search thru these snapshots, without too much efforts.
Imagine the following scenario: User comes to search for data older than 3 months. He goes to Kibana and he gets routed to the replica shard and the replica shard on the other hand "mounts" the snapshot where this data is placed.