Duplicate entries with Winlogbeat with WEF

[NightSpark (Jason McGrath)]

Hi

New to elasticsearch and have just setup Winlogbeat on a central Windows 2008 R2 server which is collecting windows events from machines using Windows Event Forwarding
I am using V 7.6.2 of elasticsearch and winlogbeat

I am finding that when I go into Kibana and into discover that view events that there are duplicate entries and that this keeps increasing over time. By duplicate I mean they have a different _id but everything else is the same, i.e timestamp, winlog event info such as id and computername. In approx 9 hours I have 60 duplicates of each entry.
I have had play around with using fingerprint processor settings and document_id in output with no luck.
My winlogbeat.yml is as follows

// 
winlogbeat.event_logs:
  - name: ForwardedEvents
    ignore_older: 72h
    
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["elastic:9200"]
  ssl.certificate_authorities: ["C:/Program Files/Winlogbeat/elasticsearch-ca.pem"]
  username: "redacted"
  password: "redacted"
  protocol: "https"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
//

[jsoriano (Jaime Soriano)]

Hey @NightSpark, welcome to discuss

Unless this is an unknown bug, there are some reasons this may happen:

• Multiple instances of Winlogbeat are running.
• Connectivity issues cause events to be sent again.

Could you please check if you have multiple instances of Winlogbeat running in your system?

To discard connectivity issues, or other issues related with the output, you can check the logs, and you can try to temporaly use the file output and check there if events are also duplicated.

[NightSpark (Jason McGrath)]

Thanks @jsoriano
I confirmed that there is only one instance of Winlogbeat running.
The file output was a good tip and I can see that there are duplicates in that output as well.
I am investigating if the source has duplicates, i.e the collected windows events though WEF.

[NightSpark (Jason McGrath)]

It is possible that subscription for forwarded events was including the collecting servers own forwarded events so it was in recursive loop.
Restricting events to a subset has seemed to fix it.