I am testing uploading Windows EVTX to Elastic using Winlogbeat and Logstash, and when opening several evtx files the Winlogbeat will not terminate the processing job, but will instead keep trying to resubmit. I originally thought it could be a connection issue between Winlogbeat and Logstash, but even when outputting to console it still repeats logs.
This is Winlogbeat 7.17.0, happens with both using environment variable on the command line and also when the evtx is coded into the config file.
Example config:
winlogbeat.event_logs:
- name: C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx
no_more_events: stop
winlogbeat.shutdown_timeout: 10s
winlogbeat.registry_file: evtx-registry.yml
output.console:
- enabled: true
codec.json:
logging:
#level: info
level: debug
files:
path: "${path.home}/winlogbeat/"
rotateonstartup: false
keepfiles: 2
The debug output shows the 2 event evtx is being repeatedly processed:
2022-02-07T11:26:12.200-0800 DEBUG [winlogbeat] beater/eventlogger.go:143 Opened successfully. {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx"}
2022-02-07T11:26:12.204-0800 INFO [winlogbeat] beater/eventlogger.go:124 Stop processing. {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx"}
2022-02-07T11:26:12.202-0800 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [5: 0, 2]
2022-02-07T11:26:12.220-0800 DEBUG [eventlog] eventlog/wineventlog.go:311 WinEventLog[C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx] Closing handle
2022-02-07T11:26:12.220-0800 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=11, end-seq=12
2022-02-07T11:26:12.220-0800 DEBUG [winlogbeat] beater/eventlogger.go:90 Successfully published events. {"id": "C:/Users/User1/Downloads/EVTX-ATTACK-SAMPLES-master/Defense Evasion/DE_Fake_ComputerAccount_4720.evtx", "event.count": 2}
2022-02-07T11:26:12.221-0800 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2
2022-02-07T11:26:12.221-0800 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2022-02-07T11:26:12.221-0800 INFO [winlogbeat] beater/winlogbeat.go:147 Shutdown will wait max 10s for the remaining 0 events to publish.
2022-02-07T11:26:12.223-0800 DEBUG [checkpoint] checkpoint/checkpoint.go:192 Checkpoint saved to disk. numUpdates=1
2022-02-07T11:26:12.235-0800 INFO [monitoring] log/log.go:192 Total metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":374,"time":{"ms":374},"value":374},"user":{"ticks":156,"time":{"ms":156}}},"handles":{"open":218},"info":{"ephemeral_id":"217722d3-96b6-4600-8df5-d321bfd6f8b0","uptime":{"ms":6425},"version":"7.17.0"},"memstats":{"gc_next":10082480,"memory_alloc":8152296,"memory_sys":18508392,"memory_total":19151696,"rss":39313408},"runtime":{"goroutines":15}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":12,"active":0,"batches":6,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":12},"read":{"bytes":0,"errors":0},"type":"console","write":{"bytes":33768,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":12,"retry":0,"total":12},"queue":{"acked":12,"max_events":4096}}},"system":{"cpu":{"cores":2}}}}}
2022-02-07T11:26:12.241-0800 INFO [monitoring] log/log.go:193 Uptime: 6.4420296s
2022-02-07T11:26:12.241-0800 INFO [monitoring] log/log.go:160 Stopping metrics logging.
2022-02-07T11:26:12.241-0800 INFO instance/beat.go:498 winlogbeat stopped.
Are there any additional configuration settings I need to configure in order for Winlogbeat to not reprocess the same log entries on repeat?