# How can I improve Logstash performance?

**URL:** <https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882>\
**Category:** Logstash\
**Created:** [July 20, 2018, 11:45am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882 "2018-07-20T11:45:06Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 20, 2018, 11:45am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/1 "2018-07-20T11:45:07Z")

</div>

Hello, I have logs in file and my `logstash.conf` looks in the way shown below. When I run `sudo ./logstash -f /path/logstash.conf -b 100000 -w 1` and monitor elasticsearch node in Kibana I see on the graph in document count that the operation of adding 123000 of rows takes 20 seconds. I have no idea why performance is so terrible. Furthermore even I have one milion of rows in my file it adds only size of one batch, in this case 123000. Is it possible to add whole files in batches to elasticsearch? In this case ten batches with 123000 of rows. The most important for me is the efficiency of adding rows to the database. I tried to use higher number of threads for instance -w equals 10 but I get error`java.lang.OutOfMemoryError: Java heap space`, so I use `sudo LS_JAVA_OPTS="-Xmx31g -Xms3g"` unfortunately it doesn't improve performance. Any ideas how can I improve performance?

```auto
input {
  file {
    path => "/run/shm/elastic/logstash/file"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {

  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}" }
  }

}
output {
    elasticsearch {
        hosts => ["address"]
        index => "logs"
        document_type => "log"
    }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 20, 2018, 11:49am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/2 "2018-07-20T11:49:37Z")

</div>

> [@white\_rabbit](#):
>
> -b 100000 -w 1

A larger batch size does not necessarily mean better performance. This looks quite high to me, so I would recommend reducing it to 1000.

Why are you limiting logstash to a single worker thread? That is surely not going to help throughput. Remove this and go with the default value in combination with a smaller batch size.

Ultimately Logstash can not send data faster than Elasticsearch can accept. What is the specification of your Elasticsearch cluster?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 20, 2018, 12:08pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/3 "2018-07-20T12:08:31Z")

</div>

I tried also for instance with `-b 1000 -w 7` but the performance was even worse.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 20, 2018, 12:15pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/4 "2018-07-20T12:15:31Z")

</div>

Is Elasticsearch running on the same node as Logstash? What throughput are you seeing? What kind of storage do you have?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 20, 2018, 12:29pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/5 "2018-07-20T12:29:30Z")

</div>

No, Elasticsearch is running on different node which is the same as the one with Logstash. Logstash is running on SSD and elasticsearch on HDD. I am not sure where I can check throughput?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 20, 2018, 12:36pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/6 "2018-07-20T12:36:12Z")

</div>

What is the specification of the node running Elasticsearch? What does CPU and disk performance metrics look like there during indexing?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 20, 2018, 12:57pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/7 "2018-07-20T12:57:08Z")

</div>

As I have mentioned before these two servers with Elasticsearch and Logstash are the same. When adding data CPU usage of more than 50% has never been achieved. Usually the usage is around 8%.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 20, 2018, 12:59pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/8 "2018-07-20T12:59:51Z")

</div>

Indexing into Elasticsearch can often be I/O intensive. What does `iostat -x` on the Elasticsearch node show during indexing? have you followed [these recommendations?](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/tune-for-indexing-speed.html)

> [@white\_rabbit](#):
>
> I am not sure where I can check throughput?

You can install X-Pack Monitoring.

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 22, 2018, 5:42pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/9 "2018-07-22T17:42:27Z")

</div>

Yes, I followed these recommendations in the past and the most optimal performance I achieved is indexing 10000 of logs per second using 30 separated processes. I mean only adding logs to elasticsearch without sending them using Logstash. When I send logs using logstash the best performance is equals to 5000 logs per second including sending and indexing in elasticsearch. Do you have an idea how can I improve performance? I monitor node using kibana and grafana. If any additional information is required just let me know. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 22, 2018, 5:52pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/10 "2018-07-22T17:52:02Z")

</div>

Can you try removing the stdout output as well? As you have a lot of cores on the Logstash machine I would try something like this: `-b 1000 -w 40`

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 22, 2018, 6:43pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/12 "2018-07-22T18:43:23Z")

</div>

What does the graph showing indexing rate show?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 22, 2018, 7:33pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/13 "2018-07-22T19:33:17Z")

</div>

Elasticsearch node grafana:

 ![es_node](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd6480d2a306976e4f7265564986815ea3771409.png)  
Logstash node grafana:  
 ![logstash_node](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c9898ed3bd45171bcaa8538f2ef213361d7ca09f.png)

---

<div class="post-metadata">

**Author:** ![lenn4rd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lenn4rd/32/33464_2.png) [@lenn4rd](https://discuss.elastic.co/u/lenn4rd)\
**Post date:** [July 25, 2018, 5:37pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/14 "2018-07-25T17:37:29Z")

</div>

The CPUs aren't maxed out which means they're not the bottleneck here. Usually using spinning hard drives on database servers as opposed to flash drives should raise a red flag. There are valid reasons to go with HDDs, e.g. better cost efficiency for big data volumes. It comes with performance penalty regarding throughput though.

Did you monitor I/O stats on the Elasticsearch instance after you started sending data from Logstash? Are there any other processes running which Elasticsearch might compete with for I/O? If that's not the cause you'd need to dig deeper, e.g. if you're running a RAID array. The way the array is set up (things like block size and the like) can have an impact on read/write performance for some use cases.

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 26, 2018, 11:41am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/15 "2018-07-26T11:41:58Z")

</div>

Now I have performance equals to 45000k per second.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 26, 2018, 11:49am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/16 "2018-07-26T11:49:10Z")

</div>

Has your Logstash config changed compared to what you posted earlier? Is all data going into a single index?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 26, 2018, 12:29pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/18 "2018-07-26T12:29:07Z")

</div>

I updated config. The current version is in the first post. Yes, all data is going into a single index.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 26, 2018, 12:52pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/19 "2018-07-26T12:52:00Z")

</div>

As you are writing into a single index and letting Elasticsearch assign document IDs I can not see any reason for this. Are all documents the same size and structure?

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 26, 2018, 12:54pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/20 "2018-07-26T12:54:55Z")

</div>

No, the length of logs can differ at least by a few characters.

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [July 26, 2018, 1:34pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/21 "2018-07-26T13:34:57Z")

</div>

How logstash adds logs to elasticsearch? Does he use bulk from helpers from elasticsearch?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 26, 2018, 1:37pm UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882/22 "2018-07-26T13:37:10Z")

</div>

Yes, Logstash uses bulk requests.

[Next page](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882.md?page=2)