Winlogbeat unable to keep up with AD security logs

[zhumphries (Zak)]

Hello.

We have recently started using winlogbeat to collect security logs from the Domain Controllers but winlogbeat is struggling to keep up with the required ingest rate. The server in question is generating approx 300-350 events per second but winlogbeat is only able to process around 250 events per second.

I have confirmed it is not a kafka output problem by using output.file and confirming the rate is approx the same. Does anyone know the expected rate that winlogbeat can pull rom the eventlog api on a well speced machine?

- name: Security
    ignore_older: 2h
    event_id: -5116
    batch_read_size: 256
    processors:
      - script:
          lang: javascript
          id: security
          file: ${path.home}/module/security/config/winlogbeat-security.js
    fields:
      topic: winlogbeat-security-dc

queue.mem:
  events: 40960

batch_read_size: Ive tried a few different values here but they dont seem to make allot of difference.

Any help would be greatly appreciated
Cheers
Zak