How can filebeat crawling and send to kafka in real time


(ruiyi luo) #1

However I optimize with various config params. There are always 3 seconds delay.
My filebeat.yml as below:

filebeat.inputs:

  • type: log
    enabled: true
    fields:
    log_topics: ngaudio_test
    paths:

    • /data/logs/action-post/*.log
      close_inactive: 2h
      backoff: 1s
      scan_frequency: 100s
      tail_files: true
      keepfiles: 7
      permissions: 0644
  • type: log
    enabled: true
    fields:
    log_topics: ngappuse_test
    paths:

    • /data/logs/app-time-use/*.log
      close_inactive: 2h
      backoff: 1s
      scan_frequency: 100s
      tail_files: true

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

queue.mem:
events: 4096
flush.min_events: 512
flush.timeout: 0s


(Noémi Ványi) #2

What is your requirement you mean by real time? Filebeat is not able to collect logs in real time, only in near real time.
You could decrease scan_frequency to 1s, so Filebeat checks for new changes every second.