# Performance issues with input-beats plugin

**URL:** <https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516>\
**Category:** Logstash\
**Created:** [December 17, 2015, 10:26pm UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516 "2015-12-17T22:26:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cooper6581](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cooper6581/32/6708_2.png) [@cooper6581](https://discuss.elastic.co/u/cooper6581)\
**Post date:** [December 17, 2015, 10:26pm UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/1 "2015-12-17T22:26:15Z")

</div>

I'm running into strange performance issues with the input-beats plug-in.

**Logstash** : 2.1.1 (duplicated on trunk yesterday FWiW)  
**input-beats** : 2.0.3  
**filebeat** : 1.0.1

It seems like throughput it being bottle-necked by something that isn't CPU / IO.

I'm doing my tests with these apache logs from NASA

My configuration is very simple, I will just replace the input with tcp / file / beats for the tests:

```
input {
    beats {
        port => 5044
    }
}

filter {
    metrics {
        meter => "events"
        add_tag => "metric"
    }
}

output {
    if "metric" in [tags] {
        stdout {
            codec => line {
                format => "Rate: %{[events][rate_1m]}"
            }
        }
    } else {
        null {
            workers => 2
        }
    }
}

```

Here is the filebeat config for reference:

```
filebeat:
  prospectors:
    -
      paths:
        - /var/tmp/issue/dataz/*
      input_type: log

output:
  logstash:
    hosts: ["localhost:5044"]

shipper:

logging:

  files:
    rotateeverybytes: 10485760 # = 10MB

```

Here are my results:

**TCP** (I just cat the file into nc): 39k/sec  
**File** : 20k/sec  
**Beats** : 3k/sec

The beats thread barely takes any CPU (~20%). I can increase throughput by adding more workers, and enabling load\_balancing on the filebeat, but this really only gets me to ~8k/sec. I also tried messing with spool\_size, and harvester\_buffer\_size with not much change.

Am I missing something obvious? If possible, I would like to get ~10k EPS for my current design. I would really like to use the lumberjack protocol instead of TCP. I haven't tested logstash-forwarder yet.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 19, 2015, 4:22am UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/2 "2015-12-19T04:22:57Z")

</div>

can you try to increase the bulk\_max\_size option in filebeat logstash output config?

```
output:
  logstash:
    bulk_max_size: ...
```

---

<div class="post-metadata">

**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\
**Post date:** [December 20, 2015, 11:35pm UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/3 "2015-12-20T23:35:29Z")

</div>

What's the CPU usage on the Logstash side? In the tests I'm doing on my laptop (with 2 cores) I also get around 8K/s, but LS seems to push the CPUs to their limits. I'll try tomorrow on a more powerful machine and report back on the results I get.

---

<div class="post-metadata">

**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\
**Post date:** [December 21, 2015, 5:43pm UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/4 "2015-12-21T17:43:17Z")

</div>

I tested on a server with 8 CPU threads, and I could get around 16 K/s by setting `bulk_max_size=3000`, and all settings set to default. I used the Logstash config that @cooper6581 posted above. I tried playing with other options (number of workers, spooler size, etc.) but nothing seemed to have an impact except for `bulk_max_size`. Filebeat CPU usage was around 40% and LS was around 150% of a CPU core.

Based on the above experiments, I would guess that the limitation is in the input-beats in Logstash. Compared to the TCP input, for example, the input-beats has to do decompression, JSON decoding (the beats -\> logstash protocol is json based) and some light data manipulation.

We'll continue to investigate this, perhaps there are some easy wins. We'll keep you up to date.

---

<div class="post-metadata">

**Author:** ![cooper6581](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cooper6581/32/6708_2.png) [@cooper6581](https://discuss.elastic.co/u/cooper6581)\
**Post date:** [January 6, 2016, 6:05pm UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/5 "2016-01-06T18:05:05Z")

</div>

This worked perfect, thanks @tudor and @steffens!

---

<div class="post-metadata">

**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\
**Post date:** [April 4, 2016, 11:13am UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/6 "2016-04-04T11:13:52Z")

</div>

Hi

I try to send 20 kB/s from filebeat to logstash and filebeat can not follow the place:

thi is my filebeat config 🙂

filebeat:  
spool\_size: 8192  
prospectors:  
-  
paths:  
- D:\zzzzz\*.log  
document\_type: mytype

output:  
logstash:  
enabled: true  
hosts: ["logstash:5043"]  
bulk\_max\_size: 8192

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/performance-issues-with-input-beats-plugin/37516/7 "2017-07-06T05:04:05Z")

</div>