How many ES nodes will be needed to handle 400GB/Day Data

Elastic Stack Elasticsearch
1

Hi Experts,

I know this is not an easy question to answer but still any idea can help here . Curious to know how many ES nodes will be sufficient to handle 400 GB/day data volume . I have gone through this blog(https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0 AND ) and this is what my requirement is

Sting Fields :- analyzed and not_analyzed
_all :- disabled
No of users :- 70+
Data Type :- Structured
Data Storage :- 30 Days
Replication factor :- 3 or 4
Search Type;- Users will be opening Dashboards , which will have lots of aggregated data .

Regards
Vikas

2

This is what I am planing to have only for ES

Total VM for ES cluster :- 8 nodes
Total RAM :- 256 GB (32 GB each node )
OS :- Linux 64 bit
Dedicates master nodes :- 3
Dedicated Hot Data Node :- 5

For Kibana I am proposing

Total VM :- 2 (6 GB RAM + 100 GB hard disk)

For LS :- 2 (8 GB RAM + 300 GB Hard Disk)

Can someone suggest if this is the acceptable architecture or do I need to modify ?

Reagrds
VG

3

These LS and Kibana are not disk hungry. You can utilise this for cold nodes or ES

4

Thanks @Ravi_Shanker_Reddy for the advice actually that is what my plan is . Moreover I want to store a copy of raw data on LS machine as well .
So here cold ES means older data ? here I have question does this mean that this data/node will never take participation during the search even if I increase the tie range ?

Also what is your recommendation on ES setup , is that architecture ok ?

Regards
VG

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.