# Increasing throughput from Filebeat to Logstash

**URL:** <https://discuss.elastic.co/t/increasing-throughput-from-filebeat-to-logstash/199450>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 13, 2019, 2:44pm UTC](https://discuss.elastic.co/t/increasing-throughput-from-filebeat-to-logstash/199450 "2019-09-13T14:44:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nathansegers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathansegers/32/79690_2.png) [@nathansegers](https://discuss.elastic.co/u/nathansegers)\
**Post date:** [September 13, 2019, 2:44pm UTC](https://discuss.elastic.co/t/increasing-throughput-from-filebeat-to-logstash/199450/1 "2019-09-13T14:44:39Z")

</div>

I am using latest versions of Filebeat, Logstash and Elasticsearch on Ubuntu 18.04 machines

I have:  
2 filebeat VM's, configured with 8 CPU cores and 16 GB Memory  
3 logstash VM's with 24 CPU cores and 64 Gb Memory (31GB Heap)  
3 Elasticsearch VM's with 16 CPU cores and 64 Gb Memory (31GB Heap)

**filebeat.yml** (same for both machines, they share a physical SSD on '/mnt/data', but each with their own allocated space and partitions)

```
filebeat.inputs:
        - type: log
          enabled: true
          paths:
              - /mnt/data/*.csv # This directory contains 502 CSV's, with a total of 780 million (780.000.000) lines and 4 columns (field_0;field_1;field_2;field_3) all of which are Integers. The directory never changes. it's historical data
          tail_files: false

queue.mem:
        events: 262144
        flush.min_events: 32768
        flush.timeout: 5s
output.logstash:
        hosts:
            - "ls-01-nathan"
            - "ls-02-nathan"
            - "ls-03-nathan"
        bulk_max_size: 32768
        loadbalance: true
        pipelining: 8
        worker: 40
http.enabled: true
monitoring.elasticsearch:
        hosts: ["es-01-nathan", "es-02-nathan", "es-03-nathan"]

```

**logstash.yml**  
(I removed all the comments and #-lines)

```
pipeline.id: ls-01-pipeline
pipeline.workers: 48
pipeline.batch.size: 131072
pipeline.batch.delay: 50
queue.type: memory
log.level: info
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://es-01-nathan:9200", "http://es-02-nathan:9200", "http://es-03-nathan:9200"]
xpack.monitoring.elasticsearch.sniffing: true

```

**logstash-config.conf**

```
input {
        beats {
                port => 5044
        }
}

filter {
        csv {
                columns => ["field_0", "field_1", "field_2", "field_3"]
                separator => ";"
        }
        mutate {
                remove_field => ["field_0", "message", "host", "@timestamp", "@version"]

                split => { "[log][file][path]" => "/" }
                split => { "[log][file][path][-1]" => "_" }
                copy => { "[log][file][path][-1][0]" => "timestamp" }

                convert => {
                        "field_1" => "integer"
                        "field_2" => "integer"
                        "field_3" => "integer"
                }
    }
    date {
            match => ["timestamp", "yyyyMMddHHmm"]
            target => "timestamp"
    }
    mutate {
            remove_field => [
            "log",
            "agent",
            "tags",
            "ecs",
            "input"]
    }
}

output {
        elasticsearch {
                hosts => ["es-01-nathan:9200", "es-02-nathan-4u:9200", "es-03-nathan-4u:9200"]
                index => "index"
        }
}

```

When I change the **filebeat** output to **output.console** and check the throughput (pv -Warl) I get around 85k/s. When I am sending the same output to Logstash, with the loadbalancing enabled. I get around 40k/s.

I have tried increasing the workers and bulk\_size, but 40k/s is the max I can get. I need to get it higher.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 1, 2019, 1:18pm UTC](https://discuss.elastic.co/t/increasing-throughput-from-filebeat-to-logstash/199450/3 "2019-11-01T13:18:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.