# Improving log ingestion speed and faster elasticsearch indexing
**URL:** https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841
**Category:** Elasticsearch
**Created:** [October 4, 2021, 6:16pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841 "2021-10-04T18:16:55Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)
#### Post date: [October 4, 2021, 6:16pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/1 "2021-10-04T18:16:55Z")
Hello,
I currently have a single node set up (a single machine where E-L-K is installed ) where a couple of gzip files are being ingested , each of these files would be of ~600MB and has several thousand lines of logs and understandably takes too much time to populate my kibana dashboards .
The current machine(single node) set up is of 16GB RAM and 100GB HDD with 8 cores, i think it may be time to set up a cluster to increase ingestion/indexing speed. I want to know if machines of above configuration would be enough to set up a machine and how many would be needed ?
And some notes on setting up a cluster would be helpful as well ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 4, 2021, 6:34pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/2 "2021-10-04T18:34:40Z")
Scaling out may help with indexing speed, but as indexing often is limited by I/O performance [I would recommend switching to SSDs](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/tune-for-indexing-speed.html#_use_faster_hardware). I would not be surprised if disk performance is your current bottleneck given that you are using spinning disks. [This video](https://www.youtube.com/watch?v=nKUpfJCBiS4) explains the benefit of using SSDs compared to HDD quite well.
---
### Author:  [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)
#### Post date: [October 4, 2021, 7:10pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/3 "2021-10-04T19:10:42Z")
Thank you @Christian_Dahlqvist for suggestion on SSD, would machines of RAM 16GB be sufficient as well ? and how many hosts would i need . My current setup is a virtual machine
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 4, 2021, 7:15pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/4 "2021-10-04T19:15:23Z")
If you look at resources around capacity planning, e.g. [this webinar](https://www.elastic.co/webinars/elasticsearch-sizing-and-capacity-planning), you will notice that indexing performance often is limited by CPU and disk I/O while the total amount of data a node can hold and serve queries for often is limited by heap size. Without knowing data volumes and retention periods it is difficult to give recommendations.
---
### Author:  [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)
#### Post date: [October 4, 2021, 7:23pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/5 "2021-10-04T19:23:15Z")
thank you 🙂 my JVM heap size is at 4GB now . I'm pasting the current index doc count and storage size that came in from 5 log files, each ~500MB. The rentention period for this data , would be for 6 hours .
---
### Author:  [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)
#### Post date: [October 4, 2021, 7:23pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/6 "2021-10-04T19:23:37Z")
And thank you for the webinar link 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 1, 2021, 7:24pm UTC](https://discuss.elastic.co/t/improving-log-ingestion-speed-and-faster-elasticsearch-indexing/285841/7 "2021-11-01T19:24:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.