Hi
We are planning a new project for a customer; we need advice on our machine calculations, if they are realistic to hold all the data.
This project consist on:
- Logstash receiving an average of 10.000 messages / second. These messages are some kind of network protocol, maximum 1500 bytes every packet. Every raw packet is processed, enriched and sent to elasticsearch. On elasticsearch, every document is 512 bytes size, aproximately (I assume 1024 bytes maximum). We have one index per day.
Documents have to be stored, at least, during 3 months. There are only few queries / day (maximum 5000)
Our guess for the server is:
-7 nodes cluster elasticsearch:
-disk: as we are storing 10.000 documents x second, this means 864.000.000 documents per day. Considering the maximum of 1kbyte per document, we need 823 GByte per day. We will have a 2 TB SSD disk to hold the daily data, and documents older than 1 day will be stored on a non-ssd 25 TB data disk.
Considering a daily index, this index could be, maximum 1 TByte per day. As I know, every shard have a limit of ¿50 Gbyte?, so we will need, at least, 20 primary shards. Is that ok?
About replicas, we are considering 6 replicas, as we have 7 nodes. Ok with that?
The 7 nodes will be located in two different cps: 3 in one cpd, 4 in the other
- cpu: 32 cpu Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz, every node
- ram: 128 GB RAM, every node
Our 7 nodes , cpu and ram numbers are derived from another production environment, which has 3 nodes and have less ingest data than this will have.
-
2 node logstash with:
- disk: 100 GB no SSD
- cpu: 32 cpu Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
- ram: 128 GB RAM
-
1 node ha-proxy to balance incoming data to the two logstash servers. We need the two logstash servers as they will be in different cpds
- disk: 100 GB no SSD
- cpu: 8 cpu Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
- ram: 16 GB RAM
What do you think about this?
Thanks a lot