# Understanding Unusual Logstash Traffic Flow Rates

**URL:** <https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821>\
**Category:** Logstash\
**Created:** [January 12, 2021, 10:59am UTC](https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821 "2021-01-12T10:59:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [January 12, 2021, 10:59am UTC](https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821/1 "2021-01-12T10:59:28Z")

</div>

Hi,  
I'd like to understand some apparently odd behaviour we're seeing with traffic hitting our logstashes vs what is being ingested into elasticsearch and s3. Note that we're using the `output isolator` pattern to decouple the two outputs from one another.

Our test pipeline is:

- 2 x filebeat
- 6 x logstash
- 6 x elasticsearch nodes (general purpose)

What we are seeing is that the logstash instances are collectively receiving/emitting around 40k msg/sec while elasticsearch and s3 are received around 13.5k msg/sec each. We have persistent queues enabled on the logstash nodes and don't see messages backing up there. And we're not seeing rejection/exceptions on the elasticsearch nodes themselves. CPU utilisation on the logstash nodes is between 20-35% and and 25-45% on the elasticsearch nodes. So nothing seems unusual.

Can I ask how logstash will report event rates when using the output isolator patten? I'm struggling to understand the discrepency between what logstash reports it is receiving/emitting vs what is being indexed to elasticsearch and written to s3.

Logstash config looks like this:

```auto
- pipeline.id: buffered-input
  queue.type: persisted
  config.string: |
    input {
      beats {
        port => 5044
      }
    }
    output {
      if [application] =~ "test" {
        pipeline {
          send_to => s3
        }
      }
      pipeline { send_to => es }
    }
- pipeline.id: buffered-out-es
  queue.type: persisted
  config.string: |
    input { pipeline { address => es } }
    output {
      elasticsearch {
        hosts => "elasticsearch:9200"
        index => "logs-%{+YYYY.MM.dd}"
        manage_template => false
        retry_max_interval => 16
        timeout => 60
        pipeline => 'test_pipeline'
        pool_max => 5000
      }
    }
- pipeline.id: buffered-out-s3
  queue.type: persisted
  pipeline.workers: 1
  pipeline.batch.size: 10000
  pipeline.batch.delay: 500
  config.string: |
    input { pipeline { address => s3 } }
    output {
      s3 {
        region => "eu-west-1"
        bucket => "test_bucket"
        time_file => 1
        rotation_strategy => "time"
        prefix => "%{application}/%{alias}/%{+YYYY}/%{+MM}/%{+dd}"
        codec => "json_lines"
        canned_acl => "private"
        encoding => "gzip"
        additional_settings => {
          "force_path_style" => true
          "follow_redirects" => false
        }
      }

```

Thx  
D

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2021, 10:59am UTC](https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821/2 "2021-02-09T10:59:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.