Winlogbeat Throughput Seems Low

Elastic Stack Beats
1

We are running a POC with beats to see if it can replace our current connector infrastructure. We can't seem to get winlogbeat to get past 1,500-2,000 eps. Our current connectors are able to peak at about 3k...Any suggestions here would be extremely helpful!

We've played around with quite a few variations on the configs, but we can't seem to get the boost.

Our effective config:

winlogbeat.event_logs:
    - name: ForwardedEvents
setup.template.settings:
  index.number_of_shards: 3
tags: ["XXXXXXXX"]
queue:
  mem:
    events: 65536
output.kafka:
  enabled: true
  hosts: XXXXXXXX 
  topic: winlogbeat
  username: XXXXXXXX
  password: XXXXXXXX
  version: 0.11.0
  worker: 1
  max_retries: -1
  ssl.enabled: true
xpack.monitoring:
  enabled: true
  elasticsearch:
    hosts: XXXXXXXX
2

You might be able to get a small bump by setting

winlogbeat.event_logs:
    - name: ForwardedEvents
      batch_read_size: 512

Last time we did a change a benchmark was done on the reader. That's without any time spent sending to an output (kafka in your case) so any numbers in a actual setup will likely be lower.

We have an open issue to try another means of reading events that might make it faster. https://github.com/elastic/beats/issues/6585

Each event log reader is independent so if you could "shard" your forwarded events across more than one event log you could have multiple readers consuming. See Creating Custom Windows Event Forwarding Logs. For example create a WEC-Powershell and WEC-System to receive events on the collector rather than dumping them all in ForwardedEvents. Then setup Winlogbeat to read from each of those logs. And see https://github.com/palantir/windows-event-forwarding/tree/master/windows-event-channels.

3

Andrew,

I, too, am looking for higher throughput windows event log consumption via winlogbeat -> kafka to replace our legacy log management system. I've made the change you recommended, and that does not make a difference. Turning on DEBUG level logging shows errors thrown and adjusting the batch read size down to 256. Other changes I've made without success:

  • added a processor to drop the message field
  • set kafka acks to 0
  • increased the max message size on kakfa broker and winlogbeat to 10M
  • increased and decreased queue sizes
  • increased and decreased flush timeout
  • tested various numbers of workers, and max_procs

Even after running winlogbeat -v -N, I'm not seeing anywhere near the ~3k messages per second rate in the benchmark: https://github.com/elastic/beats/pull/6150#discussion_r163857460.

Currently, we cannot modify our WEC setup so it does not interfere with the existing (and functioning) log shipper.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.