Average size of data indices

[m3bgwad (Mabd)]

I have more of questions regarding size of data.

• what is average of data can Elasticsearch store par second?
• if there is 1M document of data in log file and volume 1G after storing in elastic database what is the new volume? Is there over head of data adding by Elasticsearch can increase the volume?

Thank you,

[Christian_Dahlqvist]

@m3bgwad

what is average of data can Elasticsearch store par second?

That depends entirely on the size and specification of the cluster.

Have a look at the following resources:

Tune for indexing speed | Elasticsearch Guide [7.17] | Elastic

Sizing Hot-Warm Architectures for Logging and Metrics in the Elasticsearch...

@m3bgwad

if there is 1M document of data in log file and volume 1G after storing in elastic database what is the new volume? Is there over head of data adding by Elasticsearch can increase the volume?

The ratio between the size of the raw logs and the resulting index size in Elasticsearch depends on the size of the shards, index settings and the mappings used. Have a look at this section in the docs for information about how to best reduce the disk space required in Elasticsearch

[m3bgwad (Mabd)]

@Christian_Dahlqvist

That depends entirely on the size and specification of the cluster.

Have a look at the following resources

I have cluster from 3 nodes and the specs for each node is
RAM 64
CPU 18
Disk storage 1T

If I want to pull more 1M Data transaction per second in more then serves. does it have the ability to make this without happening any delays?

[Christian_Dahlqvist]

@m3bgwad

If I want to pull more 1M Data transaction per second in more then serves. does it have the ability to make this without happening any delays?

No. In order to index 1 million documents per second you will in my opinion need a considerably larger cluster with really fast local SSD disks. Wat is the average size of the documents you will be indexing?

[m3bgwad (Mabd)]

@Christian_Dahlqvist

No. In order to index 1 million documents per second you will in my opinion need a considerably larger cluster with really fast local SSD disks. Wat is the average size of the documents you will be indexing?

if you mean the disk storage is few can I make retention period each day to over come the large volume and I can increase the storage in the future as far as the disk storage is HDD this is will impact the speed in the future?

[Christian_Dahlqvist]

For high ingest throughput you will need really fast local SSD - HDD will quickly become the bottleneck. Even with SSDs you will IMHO need a significantly larger cluster. I would recommend running a benchmark with realistic data and see what you get from your current cluster.