Greetings,
I'm a student at the Hochschule Darmstadt in Germany. I'm currently working on a project for my university, where we’re trying to find a suitable Log management tool for our big data cluster.
It will be so helpful if you can provide me with some information.
Equipment:
We have 48 nodes :
28 x Dell PowerEdge C6220
2 Intel Xeon E5-2609 (4 Cores for each)
64 GB RAM
16 x 1 TB SATA 7.2 k
20 x Dell PowerEdge C6320
2 Intel Xeon E5-2620v2 (6 Cores for each)
128 GB RAM
16 x 1 TB SATA 7.2 k
The nodes are connected with a high-bandwidth and low-latency network.
Every node generate for now 500 MB of logs daily, with the total of 24 GB logs daily
The criteria we’re considering are as follows :
- The log management tool should be able to process the generated logs within 10 seconds
Generation ⇒ arrival. This means, from the logs sources to
Filebeat ⇒ Logstash ⇒ Elasticsearch ⇒ Kibana.
- Kibana UI interaction performs within 1 second
We can use 24 nodes in order to scale Logstash and Elasticsearch therefor accelerate the process.
Can the Elastic stack meet these criteria?
Are there any calculations we can do on the speed performance so if the log's quantity changed we can maintain response tau of 10 seconds?
Your help is very much appreciated.