Winlogbeat repeats batches of event logs

I am testing uploading Windows EVTX to Elastic using Winlogbeat and Logstash, and when opening several evtx files the Winlogbeat will not terminate the processing job, but will instead keep trying to resubmit. I originally thought it could be a connection issue between Winlogbeat and Logstash, but even when outputting to console it still repeats logs.

This is Winlogbeat 7.17.0, happens with both using environment variable on the command line and also when the evtx is coded into the config file.

Example config:

winlogbeat.event_logs:
  - name: C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx
    no_more_events: stop

winlogbeat.shutdown_timeout: 10s
winlogbeat.registry_file: evtx-registry.yml

output.console:
  - enabled: true
    codec.json:

logging:
  #level: info
  level: debug
  files:
    path: "${path.home}/winlogbeat/"
    rotateonstartup: false
    keepfiles: 2

The debug output shows the 2 event evtx is being repeatedly processed:

2022-02-07T11:26:12.200-0800    DEBUG   [winlogbeat]    beater/eventlogger.go:143       Opened successfully.    {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx"}
2022-02-07T11:26:12.204-0800    INFO    [winlogbeat]    beater/eventlogger.go:124       Stop processing.        {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx"}
2022-02-07T11:26:12.202-0800    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [5: 0, 2]
2022-02-07T11:26:12.220-0800    DEBUG   [eventlog]      eventlog/wineventlog.go:311     WinEventLog[C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx] Closing handle
2022-02-07T11:26:12.220-0800    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=2, start-seq=11, end-seq=12

2022-02-07T11:26:12.220-0800    DEBUG   [winlogbeat]    beater/eventlogger.go:90        Successfully published events.  {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx", "event.count": 2}
2022-02-07T11:26:12.221-0800    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:2
2022-02-07T11:26:12.221-0800    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2022-02-07T11:26:12.221-0800    INFO    [winlogbeat]    beater/winlogbeat.go:147        Shutdown will wait max 10s for the remaining 0 events to publish.
2022-02-07T11:26:12.223-0800    DEBUG   [checkpoint]    checkpoint/checkpoint.go:192    Checkpoint saved to disk. numUpdates=1
2022-02-07T11:26:12.235-0800    INFO    [monitoring]    log/log.go:192  Total metrics   {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":374,"time":{"ms":374},"value":374},"user":{"ticks":156,"time":{"ms":156}}},"handles":{"open":218},"info":{"ephemeral_id":"217722d3-96b6-4600-8df5-d321bfd6f8b0","uptime":{"ms":6425},"version":"7.17.0"},"memstats":{"gc_next":10082480,"memory_alloc":8152296,"memory_sys":18508392,"memory_total":19151696,"rss":39313408},"runtime":{"goroutines":15}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":12,"active":0,"batches":6,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":12},"read":{"bytes":0,"errors":0},"type":"console","write":{"bytes":33768,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":12,"retry":0,"total":12},"queue":{"acked":12,"max_events":4096}}},"system":{"cpu":{"cores":2}}}}}
2022-02-07T11:26:12.241-0800    INFO    [monitoring]    log/log.go:193  Uptime: 6.4420296s
2022-02-07T11:26:12.241-0800    INFO    [monitoring]    log/log.go:160  Stopping metrics logging.
2022-02-07T11:26:12.241-0800    INFO    instance/beat.go:498    winlogbeat stopped.

Are there any additional configuration settings I need to configure in order for Winlogbeat to not reprocess the same log entries on repeat?

This was a regression. It has been fixed and should ship in 7.17.1. https://github.com/elastic/beats/pull/30006

1 Like

Will this fix also be included in 8.0.1?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.