Our Current Setup
We have an elastic cloud deployment, collecting browsing logs for an e-commerce website, on a regular day our log reaches ~100GB
Given the huge amount of data, we only keep 3 days of logs then archive it.
We also set up a rollup index as 5 min intervals, grouped by url, status-code, RTT etc.
the rollup log results in ~5GB data daily
We would like to keep the rollup logs for at least 2 years
The problem is that after amount its over 100GB and did not find yet a way to automatically rollover...
So far we do a new rollup setup monthly. so we get it in a new log.
My question is
Does our setup make sense?
- Can you offer some advice on doing things better
- Is there any reference / example of a good ecomm logging strategy