Filebeat settings

[alex_96 (afsfds)]

Hello, I want to clarify if I understood the documentation correctly,
bulk_max_size - filebeat transmits events (as I understand it, let's say a line in the log file) bundled, default 50, if I set 0, then queue comes into play,
queue events - this is also the number of events (max number of events) that are also collected in packets and sent to the output, there is also flush.min_events (the minimum number of events to send) and flush.timeout (the maximum amount of time that the event waits to send)
Did I understand correctly that if bulk_max_size, queue events, flush.min_events and flush.timeout are set to 0, then there is no event queue and they are sent in real time?

[stephenb (Stephen Brown)]

Hi @alex_96

Yes looks like you basically have it correct.

If you set max_bulk_size to 0 the queue.mem settings determine the behavior.

I would ask you what you are trying to accomplish... I can tell you that that sense to set everything to 0 so that everything is "Realtime" is not a "best practice" and with any volume will actually have the opposite effect, most likely slowing down the throughput.

The defaults are pretty good if you have any scale and then typically adjust them.

Not to say you can not adjust, test try, etc... I encourage you to do that, but at scale, we are rarely if ever setting those to 0 as elasticsearch and filebeat are optimized to send bulk batches.

If you are sending to a scaled cluster increasing the workers can have a positive affect.

[alex_96 (afsfds)]

@stephenb Hello,
Thanks for the answer, it's just that I've been trying to figure out for several weeks why events arrive with a delay of a day
I will briefly describe the problem, I have 2 hosts, one with elasticsearch 7.17.5, the other with filebeat 7.17.8, ids(suricata, zeek) are installed on the host with filebeat
zeek events arrive in real time, while suricata events are ~day behind
Example:

event.original - {"timestamp":"2023-01-23T21:11:00.550666+0000
event.ingested - Jan 25, 2023 @ 11:22:47.811
@timestamp - Jan 24, 2023 @ 00:11:00.550

Everything is fine with the @timestamp, but I still can’t understand why the events reach a day later
Filebeat config:

scan_frequency: 10s
close_inactive: 15m
output.elasticsearch:
  hosts: ["https://192.168.200.20:9200"]
  username: "username"
  password: "password"
  ssl.verification_mode: "none"
  indices:
    - index: "filebeat-7.17.8-suricata-%{+yyyy.ww}"
      when.equals:
       event.module: "suricata"
    - index: "filebeat-7.17.8-zeek-%{+yyyy.ww}"
      when.equals:
       event.module: "zeek"
    - index: filebeat-7.17.8-threatintel-%{+yyyy.ww}
      when.equals:
       event.module: "threatintel"
setup.template.name: "filebeat-7.17.8"
setup.template.pattern: "filebeat-7.17.8-*"
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
bulk_max_size: 0
worker: 4
#compression_level: 9
queue.mem:
  flush.min_events: 0
  flush.timeout: 0s

Filebeat host has 4 cores(with 1 thread per core) 8 ram, 2tb disk space
If you need anything else, I'll be happy to answer.