Filebeat performance issue

[kalman]

I'm using filebeat 5.4.4 on windows to harvest from log file directly to elastic stack, the requirement is that the logs come in to elastic as soon as possible (near real-time).
for some reson the logs only show up in kibana about 5 seconds later, after the events come in the last one comes in with a additional delay of a second

my filebeat cofig:

filebeat.prospectors:
- input_type: log
  paths:
    - C:\var\log\*.log

    exclude_lines: ["^}"]
   fields:
      Host: myhost
   fields_under_root: true
filebeat.idle_timeout: 500ms
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "my-index-%{+yyyy.MM.dd}"
  pipeline: my_pipeline
  template.name: "my_template"
  template.enabled: true

[ruflin]

Have a look at the backoff variables: https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html#_backoff

[kalman]

thank you for the replay, I saw some difference in performance
Is there a recommended setting that is recommended for events to come in near real time and not make the system overload

I set the backoff to 500ms
and max_backoff to 2s

can I set max_backoff to 500ms?

[ruflin]

I would not recommend a backoff < 1s but there is no lower limit. As you mentioned correctly, the lower the number the higher the overhead. Best test it on your system on how far you can go ...