Delay betbeen timestamp and event.created on Winlogbeat

Hello!
The problem
We have unpredictable flating delay betbeen timestamp and event.created, when process logs with Winlogbeat.

Background
We use WEF to send logs from Windows host to Windows log collector. Then we use winlogbeat to send logs to ELK through Kafka topic. So, the scheme is pretty standart:
Win host -> win collector-> winlogbeat -> kafka -> logstash -> Elastic.
We have multiple collectors with same config. We use "Event Forwarding" channel to store collected events and it is near 15 Gb and store near 2 hours of logs.
So, on two of them we have floating delay between timestamp and event.created more than an hour. (Max delay that i see is 1 hour 40 min, min is 1 sec). We monitors winlogbeat\collector load and it is near 30%(RAM, CPU). So, it is enough resources. EPS on that Winlogbeat is near 1200. So, it is not to much. I read all the logs, nothing interesting, only "successfully published 1024 events".
We have another collector with 3000 EPS with same config.
Can you please help with debugging that problem? Why winlogbeat may have 1.5 hour delay and it is ok for him?

What we done:

  • Update Winlogbeat from 7.5.2 to latest 7.13.3
  • Try to disable all anivirus related soft.
  • Reboot
  • set api wineventlog-experimental
    It doesn't help at all.

Winlogbeat config:

winlogbeat.event_logs:
  - name: ForwardedEvents
    forwarded: true
  - name: HardwareEvents
    forwarded: true
tags: ["winlog"]

queue.mem:
  events: 32736
  flush.min_events: 1024
  flush.timeout: 1s
  
#================================ Outputs =====================================
output.kafka:
  hosts: ["brokers"]
  topic: 'topic'
  username: "user"
  password: "${PASS}"
  max_message_bytes: 1000000

processors:
  - add_host_metadata: ~
xpack.monitoring:
    enabled: true
    elasticsearch:
        hosts: ["myhost:9200"]

Example of delay: 1 hour, 40 min.
image
Winlogbeat metrics on same time:


Grafana metrics - 30-45% CPU (4 cores), 30-40% RAM (8 GB)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.