# Filebeat low throughput and many old files under harvesting **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 3, 2021, 8:32am UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093 "2021-02-03T08:32:40Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![aksadvance](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@aksadvance](https://discuss.elastic.co/u/aksadvance)\ **Post date:** [February 3, 2021, 8:32am UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093/1 "2021-02-03T08:32:40Z")
Hi, I am noticing this weird problem on filebeat side. Using filebeat version=7.1 I follow this setup , Filebeat -\> Logstash -\> ES. Logstash servers have been setup with persisted queue of 20GB. filebeat is reading from 2 folders each having 50+ files of 2 GB each. Most of the files were created 24 hrs back but filebeat is still harvesting these files and there is no new log lines written in these files for last 10hrs or more but don't know why filebeat is showing harvesting as below. The impact is overall throughput is less than half. We still see have half of data is still not reached ES. There is no load on logstash side and no backpressure and near empty logstash queue. I have below questions:- 1. why the throughput is so low? 2. How to see filebeat is actually sending the data and where is pointer on each file? 3. How to confirm how much data is left for streaming on the filebeat side for each file? 4. How to improve the throughput? Is too many big size files cause problem on filebeat throughput? 5. How to increase filebeat throughput ? I have also shared filebeat configs below. | 2021-02-03T06:35:50.300Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/23\_worker.log | | --- | --- | --- | --- | | 2021-02-03T06:35:50.301Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/48\_worker.log | | 2021-02-03T06:35:50.302Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/41\_worker.log | | 2021-02-03T06:35:50.302Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/28\_worker.log | | 2021-02-03T06:35:50.304Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/20\_worker.log | | 2021-02-03T06:35:50.305Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/44\_worker.log | | 2021-02-03T06:35:50.306Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/47\_worker.log | | 2021-02-03T06:35:50.307Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/50\_worker.log | | 2021-02-03T06:35:50.307Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/13\_worker.log | | 2021-02-03T06:35:50.307Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/25\_worker.log | | 2021-02-03T06:35:50.307Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/31\_worker.log | | 2021-02-03T06:35:50.307Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/52\_worker.log | ..... .... .... | 2021-02-03T06:35:50.449Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/36\_worker.log | | --- | --- | --- | --- | | 2021-02-03T06:35:50.452Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6701/37\_worker.log | | 2021-02-03T06:35:50.455Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/15\_worker.log | | 2021-02-03T06:35:50.458Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/24\_worker.log | | 2021-02-03T06:35:50.465Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/27\_worker.log | | 2021-02-03T06:35:50.468Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/34\_worker.log | | 2021-02-03T06:35:50.468Z | INFO | log/harvester.go:254 | Harvester started for file: /home/centos/ext-drive/storm/apache-storm-2.1.0/logs/workers-artifacts/topology-datumizer-priming-9-1612258878/6700/37\_worker.log | ignore\_older: 36h close\_inactive: 5m close\_removed: true clean\_removed: true queue.mem: events: 8000 flush.min\_events: 2048 flush.timeout: 0s output.logstash: # The Logstash hosts hosts: ["10.178.128.26","10.178.130.27"] loadbalance: true bulk\_max\_size: 4096 Please suggest. Regards Ajay ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [February 4, 2021, 4:36pm UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093/2 "2021-02-04T16:36:28Z")
Can you see any errors in the Logstash logs? Filebeat usually slows down when the output is not able to accept the packages from it. ---
**Author:** ![aksadvance](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@aksadvance](https://discuss.elastic.co/u/aksadvance)\ **Post date:** [February 4, 2021, 5:16pm UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093/3 "2021-02-04T17:16:31Z")
Hi, There is no errors in the logstash logs it seems fine except this below message. Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>3000, "pipeline.batch.delay"=\>10, "pipeline.max\_inflight"=\>12000, :thread=\>"#\"} [2021-02-04T13:26:24,991][WARN][logstash.javapipeline] CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 12000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 3000), or changing the number of pipeline workers (currently 4) {:pipeline\_id=\>"main", :thread=\>"#\"} There is no backpressure like 429 issue from ES. I am using logstash persisted queue and same settings and configs are working perfectly on other clusters/pipeline and queue size is going towards 20GB as filebeat throughput is high on other pipelines. But, on this pipeline this persisted queue is always below 100MB as filebeat throughput is very low, we have big size ES cluster so no back pressure from ES and logstash is always free with no errors in the log files. Regards Ajay ---
**Author:** ![aksadvance](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@aksadvance](https://discuss.elastic.co/u/aksadvance)\ **Post date:** [February 11, 2021, 5:09pm UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093/4 "2021-02-11T17:09:25Z")
@kvch can you or any member please help in debugging the issue? Or redirect me . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2021, 7:10pm UTC](https://discuss.elastic.co/t/filebeat-low-throughput-and-many-old-files-under-harvesting/263093/5 "2021-03-11T19:10:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.