I have been using Elastic Search for storing application logs.
Elastic Search version: 1.4.2
Log Retention Policy: 30 days
Number of logs generated per month: 250 million
Number of shards per index: 5
Number of replicas for index: 1
Logs/Documents in my index are not big, but number of documents are enormous.
Some Stats of existing data and extrapolation based on the same:
0.7 million - 260 MB
250 Million - 92 GB
92 GB of data per site for just application logs sounds too much to me.
So I am keen to know if these indexes or logs data under can be compressed? If so, what performance impact can it make?
My writes to elastic search will be more frequent and concurrent, while search requests will not be much frequent.