# How to ensure the integrity of filebeat output data during network fluctuations？

**URL:** <https://discuss.elastic.co/t/how-to-ensure-the-integrity-of-filebeat-output-data-during-network-fluctuations/385625>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 26, 2026, 2:03am UTC](https://discuss.elastic.co/t/how-to-ensure-the-integrity-of-filebeat-output-data-during-network-fluctuations/385625 "2026-03-26T02:03:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![h123123123](https://avatars.discourse-cdn.com/v4/letter/h/e47774/32.png) [@h123123123](https://discuss.elastic.co/u/h123123123)\
**Post date:** [March 26, 2026, 2:03am UTC](https://discuss.elastic.co/t/how-to-ensure-the-integrity-of-filebeat-output-data-during-network-fluctuations/385625/1 "2026-03-26T02:03:20Z")

</div>

The version of Filebeat I am using is 9.3.1. I set max\_retries=-1, expecting that during network fluctuations, all data can be sent to Kafka. The actual test result: data is still lost during network fluctuations. I tried changing the output to console under the same conditions, and there was no data loss; the offsets in the registry continued to increase, which indicates that Filebeat collection is functioning properly. I also tried changing required\_acks from -1 to 1, with output to Kafka, but data loss still occurred. 1、I want to know how to ensure that no data is lost.

> #=========================== Filebeat inputs =============================
> 
> filebeat.inputs:
> 
> # 模拟高可靠场景-设备端数据同步日志
> 
> # - type: log
> 
> # enabled: false
> 
> # paths:
> 
> # - /home/eventec/data\_collect/filebeatTest/high/\*.log\*
> 
> # exclude\_files: ['.swp$']
> 
> # fields:
> 
> # mqtopic: high-reliability
> 
> # botSn: F240A18010010100
> 
> # backoff: 1s # 如果扫描时没有发现新数据，进入 backoff 模式，等待 backoff 时间后再次扫描
> 
> # backoff\_factor: 1 # 每次 backoff 时间的倍数，每次等待时间 = backoff \* backoff\_factor
> 
> # max\_backoff: 1s
> 
> # close\_inactive: 10m
> 
> # # clean\_inactive: 5m # 【新增】确保文件关闭 15 分钟后，强制清除其状态，释放阻塞
> 
> # idle\_timeout: 1s
> 
> # scan\_frequency: 10s
> 
> # # ignore\_older: 2h # 【新增】忽略超过 2 小时的文件，防止处理旧日志
> 
> - type: filestream
> 
> id: high-reliability-logs
> 
> enabled: true
> 
> paths:
> 
> - /home/eventec/data\_collect/filebeatTest/high/\*.log\*
> 
> exclude\_files: ['.swp$']
> 
> fields:
> 
> mqtopic: high-reliability
> 
> botSn: F240A18010010100
> 
> backoff:
> 
> init: 1s # 对应原来的 backoff: 1s (初始等待时间)
> 
> max: 10s # 对应原来的 max\_backoff: 1s (最大等待时间)
> 
> multiplier: 2 # 对应原来的 backoff\_factor: 1 (倍增系数)
> 
> close\_inactive: 30m
> 
> scan\_frequency: 1s
> 
> take\_over:
> 
> enabled: true
> 
> #============================= Queue ===============================
> 
> queue.disk:
> 
> # 队列最大占用磁盘空间 (重要：防止占满磁盘)
> 
> max\_size: 10GB
> 
> #============================= Filebeat modules ===============================
> 
> # filebeat.registry.path: /home/eventec/fanos/filebeat-data/registry
> 
> filebeat.config.modules:
> 
> # Glob pattern for configuration loading
> 
> path: ${path.config}/modules.d/\*.yml
> 
> # Set to true to enable config reloading
> 
> reload.enabled: false
> 
> # Period on which files under path should be checked for changes
> 
> #reload.period: 10s
> 
> #----------------------------- Kafka output --------------------------------------
> 
> output.kafka:
> 
> enabled: true
> 
> # High reliability settings - retry until successful
> 
> max\_retries: -1 # Effectively infinite retries
> 
> retry\_backoff: 1s
> 
> max\_retry\_backoff: 10s
> 
> # Use acks=all for maximum reliability
> 
> required\_acks: -1
> 
> hosts: [""]
> 
> topic: '%{[fields][mqtopic]}'
> 
> compression: gzip
> 
> username: "admin"
> 
> password: "N/A"
> 
> sasl:
> 
> enabled: true
> 
> mechanism: plain
> 
> username: "admin"
> 
> password: "N/A"
> 
> key: '%{[fields.principalId]}'
> 
> partition.hash:
> 
> reachable\_only: true
> 
> # output.console:
> 
> # pretty: true
> 
> #================================ Processors =====================================
> 
> # Configure processors to enhance or manipulate events generated by the beat.
> 
> processors:
> 
> - add\_host\_metadata: ~
> 
> - add\_cloud\_metadata: ~
> 
> #============================== HTTP ============================
> 
> http:
> 
> enabled: true
> 
> port: 5066
> 
> #============================== log ============================
> 
> logging:
> 
> level: debug
> 
> metrics:
> 
> period: 1s

2、I want to know why output.events has received 108 acked (all my data) but there are still 108 pieces of data in pipeline.queue.

 ![20260326-100225](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e8d3e69ff51458064903576d98181cec7c256bb1.jpeg)