I have a two tier Java application which currently logs to disk, with daily rolling archiving. The apps collectively generate 500,000,000 (half a billion) log entries per day, 20GB compressed, or 200GB uncompressed.
The logs include apache access logs, app debug logs, custom metrics logs in csv format, and payload logs.
The horizontally scalable servers where these apps will run are one of the following sizings:
virtualized Xeon E5-2680 (v1) with 2 cores and 4GB RAM
virtualized Xeon E5-2680 (v1) with 2 cores and 8GB RAM
virtualized Xeon E5-2680 (v1) with 4 cores and 16GB RAM
How many servers should I need to run to support this load?
How much disk storage would be required to retain time based indexes for 28 days?
Is daily indexing recommended?
Have there been any comparable load tests run?