Winlogbeat 5.3.0 sends the whole content of ForwardedEvents at startup

[ssiws]

Hello,

I have a problem with Winlogbeat 5.3.0:

I configured it to push the content of the ForwardedEvents' log in elasticsearch, but each time the service is restarted, Winlogbeat sends the whole content of the ForwardedEvent in elasticsearch and this creates a lot of duplicates, etc...

When winlogbeat starts, it says that the state will be persisted to c:\programdata\winlogbeat.winlogbeat.yml : this file exists and is update automatically when a new event is pushed in ES.

The event log is limited to 1GB.

Here is the configuration file:

###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  #- name: Application
  #  ignore_older: 72h
  #- name: Security
  #- name: System
  - name: ForwardedEvents
ignore_older: 8760h

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  template.overwrite: false
  # Optional protocol and basic auth credentials.
  #protocol: "https"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

[andrewkroh]

The configuration you posted is invalid. In fact Winlogbeat won't even start.

Exiting: Error reading configuration file. 1 error: Invalid top-level key 'ignore_older' found. Valid keys are bulk_queue_size, dashboards, fields, fields_under_root, logging, max_procs, name, output, path, processors, queue_size, tags, winlogbeat accessing config

The ignore_older setting is part of the individual event_log config. So change it to:

winlogbeat.event_logs:
  #- name: Application
  #  ignore_older: 72h
  - name: ForwardedEvents
    ignore_older: 8760h

There is an open issue affecting only ForwardedEvents and restarts. See #3731. Decreasing the ignore_older time after doing the first time bulk load can help alleviate the problem, but a more effective workaround is provided in that ticket I mentioned.

[ssiws]

in fact it is a formatting error when I posted the configuration on this forum, 2 spaces was deleted inadvertently.

I tried to configure logtash as mentionned on the other thread, but the duplicates are still there, here is the config file used in logstash:

input {
  beats {
    port => 5044
  }
}

filter {
  fingerprint {
	method => "SHA1"
	key => "_id"
	source => ["@timestamp", "computer_name","log_name","record_number"]
	concatenate_sources => true
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

[andrewkroh]

In the elasticsearch output config for Logstash you have to tell it what value to use for the _id using document_id. You might also want to write the generated ID into [@metadata][id] so that it's not in the main document.

[ssiws]

Thank you for your help, it works!

[andrewkroh]

@ssiws Could you please post the working LS config. I think others will find it useful.

[ssiws]

Of course:

input {
  beats {
    port => 5044
  }
}

filter {
  fingerprint {
	method => "SHA1"
	key => "_id"
	source => ["@timestamp", "computer_name","log_name","record_number"]
	concatenate_sources => true
	target => "[@metadata][id]"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
	document_id =>"%{[@metadata][id]}"
  }
}