# Filebeat high read io when collecting log data

**URL:** <https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 1, 2022, 3:03am UTC](https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857 "2022-11-01T03:03:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zhi\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zhi_li/32/103944_2.png) [@Zhi\_Li](https://discuss.elastic.co/u/Zhi_Li)\
**Post date:** [November 1, 2022, 3:03am UTC](https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857/1 "2022-11-01T03:03:31Z")

</div>

I have a Filebeat setup to collect log data in SATA, but it does have a high read IO of around 20mb/s. below is my Filebeat config, it outputs to Kafka cluster. im just wondering is there any way to slow down the Filebeat collecting rate to reduce the read IO on disk? or any config that I can make to lower the IO?

```auto
  type: log
  encoding: plain
  scan_frequency: 3s
  paths:
    - /usr/share/filebeat/logs/*/*.log
    - /usr/share/filebeat/logs/*/*.error
  fields_under_root: true
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 5s
  tail_files: false
  symlinks: false
  backoff: 1s
  max_backoff: 10s
  backoff_factor: 2
  harvester_limit: 0
  ### Harvester closing options
  close_inactive: 20m
  close_renamed: false
  close_removed: false
  close_eof: false
  close_timeout: 0
  ### State options
  clean_inactive: 0
  clean_removed: true

```

```auto
output.kafka:
  enabled: true
  hosts: '${KAFKA_HOSTS:kafka:9092}'
  topic: 'topic'
  partition.round_robin:
    reachable_only: true

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 1, 2022, 11:47am UTC](https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857/2 "2022-11-01T11:47:59Z")

</div>

Check [this](https://discuss.elastic.co/t/filebeat-slow-improve-performance/144717/11) and [this](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuring-internal-queue)

I would try to set values next params and test:

```auto
queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

```

---

<div class="post-metadata">

**Author:** ![Zhi\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zhi_li/32/103944_2.png) [@Zhi\_Li](https://discuss.elastic.co/u/Zhi_Li)\
**Post date:** [November 2, 2022, 9:18am UTC](https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857/4 "2022-11-02T09:18:10Z")

</div>

sure, ill give it a try.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2022, 11:18am UTC](https://discuss.elastic.co/t/filebeat-high-read-io-when-collecting-log-data/317857/5 "2022-11-30T11:18:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.