# filebeat+logstash,The number of output events per second cannot break the hard limit (CPU and memory are sufficient)

**URL:** <https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 21, 2021, 1:21am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596 "2021-05-21T01:21:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![liuliugang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuliugang/32/89053_2.png) [@liuliugang](https://discuss.elastic.co/u/liuliugang)\
**Post date:** [May 21, 2021, 1:21am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596/1 "2021-05-21T01:21:45Z")

</div>

FileBeat configuration :2 core 8G Logstahs configuration :8 core 8G

FileBeat version and Logstash version 7.10.0

Single node FileBeat sends log to Logstash. The number of data per second is fixed and cannot break the 6000 limit. The CPU and memory are sufficient, but it can reach 18000 per second if it outputs to files.

All configurations have been tried following the official instructions

thanks

---

<div class="post-metadata">

**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\
**Post date:** [May 21, 2021, 3:59am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596/2 "2021-05-21T03:59:03Z")

</div>

You'll need to share your logstash.yml configuration; that's where the most relevant configuration goes regarding performance. In particular, pipeline.workers and pipeline.batch.size. I haven't changed the Java configuration options (Eg. heap size) for Logstash at all; so its largely just these two that I have changed.

For the purposes of benchmarking you'll want to avoid writing to elasticsearch; just drop the records so you can see the true limit. My own performance analytics tell me elasticsearch is easily the slowest part of my logstash pipeline (I do throw a lot of data at it).

For a data-point, one a physical server (about five years old), my statistics tell me that my current peak events/second processing is 14.4 keps (but it could go faster; logstash is not generally the bottleneck -- until I started with the memcache plugin today)

PS. I have recently started using logstash\_exporter, and I report this to Prometheus. Be sure to give each module invocation a useful 'id' field, so you can track performance of your pipeline. The logstash\_exporter just uses the REST API; here's an example (noting that the default port is 9600, not 9601 as in this example. I'm using the 'jq' tool to extract the performance counters for just a single module (which is undergoing performance engineering at present, grrrr.)

```auto
# curl -s "http://127.0.0.1:9601/_node/stats/pipelines/main" | jq '.pipelines.main.plugins.filters[] | select(.id == "networking.memcached.32")'
{
  "id": "networking.memcached.32",
  "name": "memcached",
  "events": {
    "in": 12139,
    "duration_in_millis": 1901,
    "out": 12139
  }
}

```

---

<div class="post-metadata">

**Author:** ![liuliugang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuliugang/32/89053_2.png) [@liuliugang](https://discuss.elastic.co/u/liuliugang)\
**Post date:** [May 21, 2021, 5:51am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596/3 "2021-05-21T05:51:14Z")

</div>

The Logstash output section has been replaced in the following manner  
stdout { codec =\> dots }

FileBeat profile

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0c5a389c1f2fc8c54fc7d50ed8d0df0c14385f24.png)

Logstash jvm: 4g  
Logstash configuration file  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f546640ccbb12299d7c961d5e6684c8940b3592.png)

Each log is about 1000 bytes in size

thanks

---

<div class="post-metadata">

**Author:** ![liuliugang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuliugang/32/89053_2.png) [@liuliugang](https://discuss.elastic.co/u/liuliugang)\
**Post date:** [May 24, 2021, 1:13am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596/4 "2021-05-24T01:13:59Z")

</div>

solution:  
FileBeat single section increases throughput by enabling load balancing and starting multiple Logstash nodes

Refer to the article:

> <https://github.com/elastic/beats/issues/11623>
>
> \*\*Describe the enhancement:\*\*
> \- This ER arose from a Support Ticket. User open…ed a ticket to resolve intermittent problems where filebeat processing just seemed to stop for no identifiable reason. The configuration included filebeat sending events 2 logstash processes which were publishing to kafka. 
> 
> \- what we eventually determined was that all filebeat processing was being blocked by 1 of the logstash processes which continued to process earlier events sent by filebeat. In fact logstash was sending periodic message updates back to filebeats and these messages were being reported in the filebeat log, although in a manner difficult for the user to understand. 
> 
> \- more significantly, what we also found was that a processing block in 1 logstash process caused all of the filebeat worker threads to block. This appears to have been true even though the 2nd logstash process was available for processing. 
> 
> As a result of this experience, the following enhancements to the logstash "ACK" messaging to beats have been requested: 
> 
> (1) where beats is sending to multiple logstash processes, modify the ACK protocol so that only those beat worker threads which are connected to the blocking logstash process are blocked while the worker threads connected to non-blocked logstash processes can continue working. 
> 
> (2) issue explicit messages in the beat log file explaining that specific threads are blocked waiting on logstash to complete processing of previously-sent events.   
> 
> \*\*Developer Comments From Support Ticket\*\*
> 
> \> This \[problem\] was somewhat relaxed in Beats 6.x, by making the publisher asynchrouns and removing the spool as is. Still Filebeat needs processing ACK to be done in order for keeping the registry file in a sane state. That is event Filebeat 6.x might suffer similar issues here.
> 
> \>Filebeat MUST keep order of events when writing the registry file. As filebeat 6.x supports out-of-order batch publishing, all state updates need to be kept in memory in filebeat.
> If a batch is never ACKed, then filebeat would require to accumulate all state in memory, eventually going OOM. This is already supported in the small, but at some point in time the queue of state updates is full. This is where even Filebeat 6.x would start to block.
> 
> \> one way to resolve it is:
> 
> \> add a timeout to output workers, duplicating batches to other idle workers once the timeout kicks in. Once a batch is ACKed by one output worker, the other outputs will be cancelled/reset stopping to processing of the current Batch.
> 
> \>The change in the LB algorithm creates a few events that must be locked:
> 
> \> - resend timeout kicks in
> \> - one worker finally ACKing a duplicate batch
> \> - Batch being cancelled
> \> - followup event: cancell success or ACK received (duplicate events).
> \> - The logger/workers are alrready aware of the endpoint. So the actual worker+endpoint will be included in these log messages.
> 
> \> This strategy guarantees progress and is often used to maximize throughput in a dynamically load-balanced system, in case some services sees some slow-down.
> The disadvantage is the potential for duplicates, but filebeat already has at least once semantics. So this is no change in semantics at all. If an output missbehaves or an IO error occurs we have to send again.
> 
> \*\*Related Issues/Enhancement Requests\*\*
> 
> https://github.com/elastic/apm-server/issues/1298
> https://github.com/elastic/beats/issues/8080
> https://github.com/elastic/beats/pull/7925

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 21, 2021, 3:14am UTC](https://discuss.elastic.co/t/filebeat-logstash-the-number-of-output-events-per-second-cannot-break-the-hard-limit-cpu-and-memory-are-sufficient/273596/5 "2021-06-21T03:14:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.