Winlogbeat 7.5.2 duplicate events

[knrdv]

Issue: One event from one Windows host shows like multiple identical events in Kibana. Events shown in Kibana differentiate only by the "_id" field with everything else being identical. What could be the problem?
Verisons: Running Winlogbeat, Logstash, Elasticsearch and Kibana on version 7.5.2.

[kvch (Noémi Ványi)]

Could you please share Winlogbeat and Logstash configuration and format it using </>?

[knrdv]

Hi, yes, here is the Winlogbeat.yml config:

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["<hostname_redacted>:5045"]
  ssl.enabled: true

Logstash configuration is configured using pipelines.yml file which looks like this:

- pipeline.id: filebeat
  path.config: "/etc/logstash/conf.d/filebeat/*.conf"

- pipeline.id: winlogbeat
  path.config: "/etc/logstash/conf.d/winlogbeat/*.conf"

pipelines.yml pulls winlogbeat configuration from two .conf files, the first winlogbeat .conf file defines inputs and it's called 01-input.conf and it looks like this:

input {
  beats {
    port => 5045
    ssl => true
    ssl_key => '/usr/share/logstash/config/server.key'
    ssl_certificate => '/usr/share/logstash/config/server.crt'
    ssl_verify_mode => 'none'
  }
}

The second file represents logstash output configuration for winlogbeat pipeline and it's called 03-output.conf:

output {
  elasticsearch {
    hosts => ["https://<hostname_redacted>:443"]
    user => "<username_redacted>"
    password => "<password_redacted>"
    manage_template => false
    ssl_certificate_verification => false
    ssl => true
    cacert => "/usr/share/logstash/config/<certificate_name_redacted>.pem"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Hope this helps.

[knrdv]

Hi, we still haven't found a solution to this problem. We could provide more info about the configuration if that might help?