# Elasticsearch/Logstash low indexing rate
**URL:** https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852
**Category:** Elasticsearch
**Created:** [June 12, 2020, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852 "2020-06-12T08:48:24Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@urkole](https://discuss.elastic.co/u/urkole)
#### Post date: [June 12, 2020, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/1 "2020-06-12T08:48:24Z")
Hello,
I'm setting up an Elasticsearch cluster in my laptop to do some testing before migrating to a better infrastructure, so I'm still learning how this works.
As I've said, I have an Elasticsearch instance running (one node, one shard, no replication). I'm reading log files from Logstash and outputting them to the ES instance with the most basic pipeline:
```auto
input {
file {
mode => "read"
path => "/Users/urko/tests/elastic/log-samples/workdata/*"
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "test_raw_2"
}
}
```
My log file is just some sample Apache Httpd 10GB log file (136736170 lines). I've noticed that when ingesting this data into ES, the ingestion rate is really low, as it takes around 5h before it finishes. I've also tried splitting the file into 10 1GB files to see wether that could speed things up, but that didn't help.
# My specs
## Laptop
[MacBook Pro (Early 2015)](https://everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-i7-3.1-13-early-2015-retina-display-specs.html)
3.1 GHz Dual-Core Intel Core i7
16 GB 1867 MHz DDR3
macOS v10.15.4
## Elasticsearch configuration
I didn't specify anything in particular except the JVM heap size, as I read that it's important. It has 8GB assigned, which is half of my systems total memory.
## Logstash configuration
After reading [logstash.yml](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html) and [Performance Troubleshooting](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html) I've tried with multiple heap sizes (2GB and 4GB), and I've found no difference performance-wise.
# Question
So I'm guessing that 2GB/h is a slow ingestion rate and that something in my setup is wrong. I've noticed (based on the number of documents in my index) that it gets slower the longer the process has been running (maybe some GC issue? no idea).
Elasticsearch and Logstash outputs show no errors or warnings, and I haven't experienced any crash either.
Can I get any help? What's making this run so slow? Is Logstash not supposed to run alongside Elasticsearch?
I can provide additional information if required.
Thanks in advance.
---
### Author:  [@urkole](https://discuss.elastic.co/u/urkole)
#### Post date: [June 12, 2020, 2:57pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/2 "2020-06-12T14:57:29Z")
After doing some tampering with the JVM configuration (both Elasticsearch's and Logstash's) and trying multiple configuration, I can't see any improvement and I don't think my hardware is the cause of the bottleneck. It's still taking around 30min/GB and the CPU and Heap usage seem Ok to me (see images below).
So, what's causing the ingestion rate to be so slow? Maybe some configuration that I'm unaware of?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 12, 2020, 3:00pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/3 "2020-06-12T15:00:35Z")
With Elasticsearch the bottleneck is often disk I/O and disk utilisation. Are you monitoring this?
---
### Author:  [@urkole](https://discuss.elastic.co/u/urkole)
#### Post date: [June 15, 2020, 6:53am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/4 "2020-06-15T06:53:32Z")
I don't think that's the case either. Here's the output from `iostat -w 3` during the process: [link](https://drive.google.com/file/d/19YQU0wi5nSwJsslF3pXXXGJltxbCDOop/view?usp=sharing).
The MB/s column is around 10MB/s most of the time, and sometimes goes up to 100MB/s~. I've seen this at around 400 when doing a `cp` operation on large files.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 15, 2020, 7:02am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/5 "2020-06-15T07:02:49Z")
Can you please run iostat -x so we can see disk utilization and iowait?
---
### Author:  [@urkole](https://discuss.elastic.co/u/urkole)
#### Post date: [June 15, 2020, 8:19am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/6 "2020-06-15T08:19:00Z")
MacOS `iostat` doesn't have a `-x` option.
Anyways, I've already finished my tests and was just asking to see wether I was doing something wrong.
Should I mark this issue as solved? If so, which answer is the solution?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 13, 2020, 8:19am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-low-indexing-rate/236852/7 "2020-07-13T08:19:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.