How is log storage working?

Elastic Stack Logstash
1

I have an ELK server where all my other (50+) servers are sending their logs to it. It's a virtual machine and i gave it at first 60GB of storage which i divided.

I monitor this server and i received a warning that my harddisk space is running out of space. The ELK server runs for 3 weeks now so it is using a lot of space to store the logs.

I would like to understand how the storage is working. I know it makes an index of all the logs, but does it also (g)zips the logs?

I would like to have the logs for at least 6 months and after that the oldest ones can be deleted. How is ELK working with these kind of settings?

Can anyone give me some idea about this?

2

The relation between the size of the data being indexed and how much space it takes up on disk will depend a lot on the mappings you use. The default Logstash config indexes most fields both as analyzed and not_analyzed which adds a lot of flexibility when you query it, but tends to take up a fair bit of space on disk. You can save a significant amount of disk space by optimising how you store data and map it. We published a blog post. last month which shows the effect eliminating certain fields and rationalising mappings can have on the size for typical logging use cases.

Best regards,

Christian