# Suggestions - Hardware and software configurations of Logstash, Elasticsearch, Kibana

**URL:** <https://discuss.elastic.co/t/suggestions-hardware-and-software-configurations-of-logstash-elasticsearch-kibana/34689>\
**Category:** Elasticsearch\
**Created:** [November 16, 2015, 2:50pm UTC](https://discuss.elastic.co/t/suggestions-hardware-and-software-configurations-of-logstash-elasticsearch-kibana/34689 "2015-11-16T14:50:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![shankar-es](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@shankar-es](https://discuss.elastic.co/u/shankar-es)\
**Post date:** [November 16, 2015, 2:50pm UTC](https://discuss.elastic.co/t/suggestions-hardware-and-software-configurations-of-logstash-elasticsearch-kibana/34689/1 "2015-11-16T14:50:52Z")

</div>

# what we want to setup is :

- **_Kafka server =\> Kafka-logstash-input plugin =\> Elasticsearch =\> Kibana_**

- Use =\> Real-time analysis

- Analysis frequency =\> last 1 week data

# Below are the our Kafka message system details :

- Message rate for kafka server =\> 30000 records / seconds
- Message format =\> json
- Mesage Size =\> 5Kb / record
- Expected daily data size =\> 500 GB

# Could you please give us some hardware and configuration suggestions?. here are the some:

- AWS instances hardware configurations.
- Memory (RAM) and CPU Tuning configurations for all software's like Kafka-logstash-input plugin =\> Elasticsearch =\> Kibana
- Hard Disk requirements.
- Data compression Methods.
- Other configurations like sharding, replicas etc..

---

<div class="post-metadata">

**Author:** ![angad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/angad/32/5848_2.png) [@angad](https://discuss.elastic.co/u/angad)\
**Post date:** [November 17, 2015, 3:50am UTC](https://discuss.elastic.co/t/suggestions-hardware-and-software-configurations-of-logstash-elasticsearch-kibana/34689/2 "2015-11-17T03:50:14Z")

</div>

With elasticsearch its best to experiment with hardware as requirements may vary with what type of content you are indexing, number of fields, query rate, shards, replicas.

For me - Currently using 20 nodes for 350GB per day of log messages with over 1000 different fields (each message contains upto 20-30 fields). We store data for the past 90 days but only have open indices for past 10 days.  
each node is a physical box with 64gb RAM, 2x3TB RAID 0 disks, 12 cores.  
number of shards - 20  
number of replicas - 3

We dont use kafka - but use logstash-forwarder and logstash for receiving and processing logs.

Would be interesting to hear other setups of similar or larger scale.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:38pm UTC](https://discuss.elastic.co/t/suggestions-hardware-and-software-configurations-of-logstash-elasticsearch-kibana/34689/3 "2017-07-05T23:38:04Z")

</div>