Disk space measure for Elasticsearch service

[kaushalshriyan]

Hi,

I have provisioned 300 GB of Hard disk storage to Elastic search stack. Is there a way to measure how much storage is consumed by ES service, as I need to work on Capacity planning. For example, how much memory and storage to be provided to ES service? I am pushing all the application logs to Elastic Stack. How many logs data can be stored in Elasticsearch service for a 300 GB of Hard disk storage?

Please guide me. Thanks in advance.

Best Regards,

Kaushal

[Rios]

It's not clear specification for a simple answer. You might have <300 GB disk for indices because:

• 1000 bytes=1 KB on the physical disk and 1024 bytes=1 KB on the OS level will be ~280 GB
• you may have ES logs, and temp files on the same partition
• will use replica or not
• disk space watermarking
• ES has own system indices

Let's simplify, you have all 300.00 GB only for indices and 1 replica for FT with 1 acceptable node down.

• Disk watermarking (AZ-5 ) - 10-15%, don't count below 10%=> 270 GB
• 1 GB data in logs or pure JSON ready for ES insert is not 1 GB on disk. This is related to the Lucene engine and ES indexing. Also, it's possible to use the compression which give more space and reduce performances, roughly 10% =>240 GB
• Replica shards minimum is 1 for fault tolerance - 50% => 120-125 GB

In total you will have ~120 GB for indices. Not great not terrible © Dyatlov

Plan to use:

• data only on the dedicated disk
• use ILM for data retention, this will give option for fine tuning indices disk space
• replica 0, for acceptable data loss
• compression, deflate where is possible
• snapshots repository on a separated disk
• nodes monitoring

If you like we can provide you with links to read about text above. then go for design.