what we want to setup is :
Kafka server => Kafka-logstash-input plugin => Elasticsearch => Kibana
Use => Real-time analysis
Analysis frequency => last 1 week data
Below are the our Kafka message system details :
- Message rate for kafka server => 30000 records / seconds
- Message format => json
- Mesage Size => 5Kb / record
- Expected daily data size => 500 GB
Could you please give us some hardware and configuration suggestions?. here are the some:
- AWS instances hardware configurations.
- Memory (RAM) and CPU Tuning configurations for all software's like Kafka-logstash-input plugin => Elasticsearch => Kibana
- Hard Disk requirements.
- Data compression Methods.
- Other configurations like sharding, replicas etc..