Winlogbeat infinite loop through evtx file and shipping duplicate events when manually shipping events from an evtx file. Starting v7.16.0 to v7.17

[Bryan_Hamilton]

Hi, since winlogbeat 7.16.0, when trying to manually ship logs, the beat loops through the evtx file and keeps resending duplicate events until you hit ctrl+C to stop it regardless of the setting no_more_events: stop. I started noticing this behavior when 7.16.0 came out so kept using 7.15.2 every time I had to manually ship logs, thinking the problem might get fixed in a future release but it has been a few month and the behavior on new releases is the same so I am thinking maybe an change in the new binaries require a change in the config file?

To illustrate this problem, I have an evtx file with only 4 events in it and am trying to ship it elastcsearch through logstash. I am starting the test with the latest winlogbeat 7.17.0. Here is my directory structure:

The config file used to manually ship the logs is the file called winlogbeat-evtx.yml and contains the following configurations:

# ======================== Winlogbeat specific options =========================

winlogbeat.event_logs:
  - name: ${EVTX_FILE}
    tags: ["test_data"]
    no_more_events: stop
    processors:
      - script:
          when.equals.winlog.channel: Security
          lang: javascript
          id: security
          file: ${CWD}/module/security/config/winlogbeat-security.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
          lang: javascript
          id: sysmon
          file: ${CWD}/module/sysmon/config/winlogbeat-sysmon.js
      - script:
          when.equals.winlog.channel: Windows PowerShell
          lang: javascript
          id: powershell
          file: ${CWD}/module/powershell/config/winlogbeat-powershell.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
          lang: javascript
          id: powershell
          file: ${CWD}/module/powershell/config/winlogbeat-powershell.js

winlogbeat.shutdown_timeout: 1s
winlogbeat.registry_file: "${CWD}/log_and_registry/evtx-registry.yml"

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  enabled: true
  # The Logstash hosts
  hosts: ["logstashhost.fqdn:5360"]
  loadbalance: true
  
  # Enable SSL
  ssl.enabled: true

  # List allowed SSL/TLS versions
  ssl.supported_protocols: [TLSv1.2, TLSv1.3]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["${CWD}/certs/ca_certname.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "${CWD}/certs/certname.crt"

  # Client Certificate Key
  ssl.key: "${CWD}/certs/certname.key"
  
  # Server Certificate verification mode: full, strict, certificate or none
  ssl.verification_mode: strict

Note that the no_more_events setting is set to stop.

Now from powershell:

1. Change directory to the manual_evtx_shipping directory:
   cd xxxx\manual_evtx_shipping
2. Process the evtx file:
   .\winlogbeat-17.17.0\winlogbeat.exe -e -c .\winlogbeat-17.17.0\winlogbeat-evtx.yml -E CWD="C:\xx\xx\manual_evtx_shipping\winlogbeat-17.17.0" -E EVTX_FILE="C:\xx\xx\manual_evtx_shipping\evtx_files\test_file.evtx"

When I run this command, the command keeps running until I hit Ctrl+C to manually stop it. In kibana, I see as many duplicate events as it was able to send until I hit Ctrt+C (this file only has 4 events).

Now I replace only the winlogbeat.exe binary in the winlogbeat-7.17.0 folder with with the binary from 7.15.2 (timestamped November 4th 2021)

I delete all the previous events from kibana. With everything else the same, I run the above processing powershell command, the command runs for a second or two and stops by itself. In kibana, I have the expected events: No duplicates.

I have ran this test with every binary released after 7.15.2, with the same config file, they all produce duplicate events but binaries from 7.15.2 and below do not produce duplicate events.

Is this a bug? or do I need to update the config file with a particular setting?