# Filebeat not sending most logs across the network **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 20, 2024, 12:41am UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472 "2024-01-20T00:41:15Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 24, 2024, 5:46pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/21 "2024-01-24T17:46:27Z")
I took a packet capture and saw some traffic on port 5044, but I'm not able to decode it visually. We're not using SSL, but it's not plain text either. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 24, 2024, 5:59pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/22 "2024-01-24T17:59:22Z")
The traffic is compressed, which you can disable on the output using `compression_level: 0` [Logstash output settings for filebeat](https://github.com/elastic/beats/edit/8.12/libbeat/outputs/logstash/docs/logstash.asciidoc) Loading the capture into wireshark should allow you to view the compressed body or you can disable compression which should show the body in tcpdump ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 24, 2024, 7:05pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/23 "2024-01-24T19:05:20Z")
We turned off compression and it turns out that the traffic _is_ making it across the wire to our Logstash servers. I guess my ability to understand `tcpdump` while under pressure isn't as good as I thought it was. We still don't understand why _some_ traffic is being processed by Logstash, but not _all_ traffic. Logs of certain types are being processed for some servers, but not all. The servers that are having issues with a specific log file being processed are not having issues with other log files being processed. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 24, 2024, 7:14pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/24 "2024-01-24T19:14:07Z")
You can use the Logstash APIs to pull pipeline stats [Node Stats API | Logstash Reference [7.17] | Elastic](https://www.elastic.co/guide/en/logstash/7.17/node-stats-api.html) Each Pipeline has stats for events failed, as well as events in and out of each plugin which should help you figure out what might be happening with the events. You're welcome to post the node stats here and I can take a look but it might be difficult for me to help without the pipeline definitions. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 24, 2024, 8:30pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/25 "2024-01-24T20:30:14Z")
It's interesting, I have tried to tweak the batch size and number of workers for my pipelines, but the API says it's the default and doesn't report per-pipeline data. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 24, 2024, 8:32pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/26 "2024-01-24T20:32:53Z")
Have you identified via the `curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'` API which pipeline or which pipeline steps appear to be dropping events? ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 24, 2024, 9:04pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/27 "2024-01-24T21:04:18Z")
I can see `failures` though, nothing specifically about drops. The particular section that I know should be handling traffic that we're not seeing all of does not have any failures. The entire pipeline has a difference of only about 5k between in and out, on a 32 core machine, which I assume is trivial. Especially since the 3 Logstash servers we're running in that DC are processing over 80M log lines per hour. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 24, 2024, 9:08pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/28 "2024-01-24T21:08:38Z")
I would expect to see failures represented in the logs of the logstash server, are you able to see any errors on the logstash logs? ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 24, 2024, 11:19pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/29 "2024-01-24T23:19:58Z")
You'll want to look at the in and out of each plugin in the pipeline to see which plugin is dropping or filtering the messages ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 25, 2024, 6:32pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/30 "2024-01-25T18:32:24Z")
We figured out that in 1 of the DCs we're having issues, it's because a particular log had its verbosity turned up to 11, so the pipeline was getting clogged. So that's 1 mystery solved. We may be seeing a similar situation in our other DC that's having issues, but I'm not sure how to determine what that clog is. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 25, 2024, 7:17pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/31 "2024-01-25T19:17:32Z")
I would recommend considering using the Logstash integration with Elastic Agent to monitor your Logstash clusters: [Logstash | Documentation](https://docs.elastic.co/integrations/logstash) You may also consider looking in Elasticsearch to see volumes of data coming from each DC to identify any other high volume DCs. You could also implement something like Network Packet Capture integration in elastic agent or using Packetbeat on the Logstash node (or other network-logging) to better understand where your traffic is coming from. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [January 26, 2024, 10:05pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/32 "2024-01-26T22:05:08Z")
The issue in our other DC seems to be a capacity one. We had 4 lightly loaded servers, so we decommissioned 1. The other 3 were still lightly loaded, plenty of free RAM and CPU headroom, but for some reason the system couldn't handle the requirements and logs were not able to be processed in a timely manner. We have increased both `pipeline.workers` and `pipeline.batch.size` to the point where we were not seeing any additional performance gains, but without the 4th server we were unable to squeeze any more performance out of the system. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 27, 2024, 1:15am UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/33 "2024-01-27T01:15:00Z")
It might be worth looking at the pipeline steps being performed, I think the stats give you info on step throughput to see where the bottleneck is in the pipeline. You might be able to remove or optimize any heavy pipeline steps. It may also be worth investigating if the bottleneck is actually logstash writing to elasticsearch which you may be able to optimize as well. It might also make sense to see if you can tune the beats max bulk size to match your logstash batch size as well. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://discuss.elastic.co/u/reshippie)\ **Post date:** [February 1, 2024, 6:34pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/34 "2024-02-01T18:34:34Z")
I don't know if I should start a new thread, but we're trying to squeeze more performance out of our Logstash servers. When we were having issues, the CPU load did not go up, neither did the RAM usage. The hardware we use has 16 or 32 cores and 64GB of RAM, with 24GB allocated for Java heap space. We have tuned the `pipeline.workers` to 128 and the `pipeline.batch.size` to 500, but we can't get the CPU load on any of them to go above 8. Adding an additional Logstash server to the pool resolved our ingest issue, so we don't believe the issue is write performance, but we don't understand where our bottleneck is. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [February 1, 2024, 7:16pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/35 "2024-02-01T19:16:01Z")
Are you using persistent queues or any other settings that might affect performance? Do you have a complicated pipeline? You can look at the duration\_in\_millis in the pipeline stats for your pipeline to identify which steps are taking the most time/CPU in your pipeline. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 29, 2024, 9:16pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472/36 "2024-02-29T21:16:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/filebeat-not-sending-most-logs-across-the-network/351472.md?page=1)