# Logstash pipeline metric events not output under load

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451>\
**Category:** Logstash\
**Created:** [February 7, 2019, 12:46pm UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451 "2019-02-07T12:46:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mario\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_wood/32/88001_2.png) [@mario\_wood](https://discuss.elastic.co/u/mario_wood)\
**Post date:** [February 7, 2019, 12:46pm UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451/1 "2019-02-07T12:46:17Z")

</div>

I have a pipeline configured with logstash 6.3.0 which should capture metrics using the metrics filter for the number of events then output them to statsd. My pipeline config is as follows (i've removed some bits of company specific config):

```
input {
  tcp {
    id => "tcp_input"
    port => 3515
    codec => 'json_lines'
    tags => ["tcp"]
  }
  http {
    id => "http_input"
    port => 8080
    tags => ["http"]
  }
  heartbeat {
    id => "heartbeat_input"
    add_field ...
    ...
    interval => 10
  }
}
filter {
  if "_jsonparsefailure" in [tags] { drop { } }

  if conditional 1 { drop { } }

  if conditional 2 { drop { } }

  if conditional 3 { drop { } }

  truncate {
    id => "truncate_large_message_field"
    fields => ["Message", "message"]
    length_bytes => 10240
  }

  mutate {
    id => "main_mutate_filter"
    rename => ...
    ...
    rename => ...
    add_field => ...
    add_field => ...
  }

  metrics {
    id => "logstash_metrics"
    meter => ["events"]
    add_tag => "metric"
    flush_interval => 10
    rates => [1]
  }
}
output {
  if "metric" not in [tags] {
    tcp {
      host => "<%= logstash_address %>"
      port => 3515
      codec => "json_lines"
    }
  }

  if "metric" in [tags] {
    statsd {
      id => "statsd_output"
      gauge => { "events.rate_1m" => "%{[events][rate_1m]}" }
      count => { "events.count" => "%{[events][count]}" }
      port => "8125"
      host => "<%= monitoring_address %>"
      namespace => "<%= tenant %>"
      sender => "<%= feature %>_<%= position %>"
    }
  }
}

```

Without any load I see the metrics in our monitoring system, but as soon as I start pushing any load at logstash I don't see the metrics.

I have log.level: debug set in logstash.yml and I can see that my statsd output is not being used under load.

I have 3 instances behind an ELB on AWS, currently c5.large, but I'm testing with other instance types.

Any help would be much appreciated.

Thanks  
Andy

---

<div class="post-metadata">

**Author:** ![mario\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_wood/32/88001_2.png) [@mario\_wood](https://discuss.elastic.co/u/mario_wood)\
**Post date:** [February 15, 2019, 8:54am UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451/2 "2019-02-15T08:54:11Z")

</div>

For reference, this is what we're seeing for our received metrics:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/5032316134a9a1cbab2464a6ab9f5b633d580cb4.png)

---

<div class="post-metadata">

**Author:** ![mario\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_wood/32/88001_2.png) [@mario\_wood](https://discuss.elastic.co/u/mario_wood)\
**Post date:** [February 26, 2019, 5:49pm UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451/3 "2019-02-26T17:49:10Z")

</div>

I've figured this out, it's because we have a high batch size (pipeline.batch.size: 3000). I guess what's happening is that under low load there's no batching of the received events then as load increases, the events start to get batched, but there are insufficient metrics events to be batched and sent. I've tested reducing the batch size and the metrics come through as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 26, 2019, 5:49pm UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451/4 "2019-03-26T17:49:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.