I think about usage of ElasticSearch for logging.
System generates about 5-10k log entries per second, average, total size is about 500 Mb per second. Most of log entries are small, but some log entries have size in range 300-500 kb. Such log entries have data in xml format. In case of growth of load, logging subsystem should be able to handle it, if add more servers.
Time to store logs in Elastic is undefined yet. I thinking about 1 month, but it is possible to set less duration, like 2 weeks.
So, logging system should be able to efficiently compress data, to not requires lots of drives, and be able to handle the amount of data.
Can ElasticSearch handle the amount of data efficiently, it is suitable for case when log entries have big size?