Disk space measure for Elasticsearch service

Hi,

I have provisioned 300 GB of Hard disk storage to Elastic search stack. Is there a way to measure how much storage is consumed by ES service, as I need to work on Capacity planning. For example, how much memory and storage to be provided to ES service? I am pushing all the application logs to Elastic Stack. How many logs data can be stored in Elasticsearch service for a 300 GB of Hard disk storage?

Please guide me. Thanks in advance.

Best Regards,

Kaushal

1 Like

It's not clear specification for a simple answer. You might have <300 GB disk for indices because:

  • 1000 bytes=1 KB on the physical disk and 1024 bytes=1 KB on the OS level will be ~280 GB
  • you may have ES logs, and temp files on the same partition
  • will use replica or not
  • disk space watermarking
  • ES has own system indices

Let's simplify, you have all 300.00 GB only for indices and 1 replica for FT with 1 acceptable node down.

  • Disk watermarking (AZ-5 :slight_smile:) - 10-15%, don't count below 10%=> 270 GB
  • 1 GB data in logs or pure JSON ready for ES insert is not 1 GB on disk. This is related to the Lucene engine and ES indexing. Also, it's possible to use the compression which give more space and reduce performances, roughly 10% =>240 GB
  • Replica shards minimum is 1 for fault tolerance - 50% => 120-125 GB

In total you will have ~120 GB for indices. Not great not terrible © Dyatlov

Plan to use:

  • data only on the dedicated disk
  • use ILM for data retention, this will give option for fine tuning indices disk space
  • replica 0, for acceptable data loss
  • compression, deflate where is possible
  • snapshots repository on a separated disk
  • nodes monitoring

If you like we can provide you with links to read about text above. then go for design.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.