Disk space measure for Elasticsearch service

Elastic Stack Elasticsearch
1

Hi,

I have provisioned 300 GB of Hard disk storage to Elastic search stack. Is there a way to measure how much storage is consumed by ES service, as I need to work on Capacity planning. For example, how much memory and storage to be provided to ES service? I am pushing all the application logs to Elastic Stack. How many logs data can be stored in Elasticsearch service for a 300 GB of Hard disk storage?

Please guide me. Thanks in advance.

Best Regards,

Kaushal

2

It's not clear specification for a simple answer. You might have <300 GB disk for indices because:

  • 1000 bytes=1 KB on the physical disk and 1024 bytes=1 KB on the OS level will be ~280 GB
  • you may have ES logs, and temp files on the same partition
  • will use replica or not
  • disk space watermarking
  • ES has own system indices

Let's simplify, you have all 300.00 GB only for indices and 1 replica for FT with 1 acceptable node down.

  • Disk watermarking (AZ-5 :slight_smile:) - 10-15%, don't count below 10%=> 270 GB
  • 1 GB data in logs or pure JSON ready for ES insert is not 1 GB on disk. This is related to the Lucene engine and ES indexing. Also, it's possible to use the compression which give more space and reduce performances, roughly 10% =>240 GB
  • Replica shards minimum is 1 for fault tolerance - 50% => 120-125 GB

In total you will have ~120 GB for indices. Not great not terrible © Dyatlov

Plan to use:

  • data only on the dedicated disk
  • use ILM for data retention, this will give option for fine tuning indices disk space
  • replica 0, for acceptable data loss
  • compression, deflate where is possible
  • snapshots repository on a separated disk
  • nodes monitoring

If you like we can provide you with links to read about text above. then go for design.