# Winlogbeat 5.3.0 sends the whole content of ForwardedEvents at startup

**URL:** <https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596>\
**Category:** Beats\
**Tags:** winlogbeat\
**Created:** [March 30, 2017, 6:42am UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596 "2017-03-30T06:42:09Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssiws](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@ssiws](https://discuss.elastic.co/u/ssiws)\
**Post date:** [March 30, 2017, 6:42am UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/1 "2017-03-30T06:42:09Z")

</div>

Hello,

I have a problem with Winlogbeat 5.3.0:

I configured it to push the content of the ForwardedEvents' log in elasticsearch, but each time the service is restarted, Winlogbeat sends the **whole** content of the ForwardedEvent in elasticsearch and this creates a lot of duplicates, etc...

When winlogbeat starts, it says that the state will be persisted to c:\programdata\winlogbeat.winlogbeat.yml : this file exists and is update automatically when a new event is pushed in ES.

The event log is limited to 1GB.

Here is the configuration file:

```
###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  #- name: Application
  # ignore_older: 72h
  #- name: Security
  #- name: System
  - name: ForwardedEvents
ignore_older: 8760h

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  template.overwrite: false
  # Optional protocol and basic auth credentials.
  #protocol: "https"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 30, 2017, 3:47pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/2 "2017-03-30T15:47:48Z")

</div>

The configuration you posted is invalid. In fact Winlogbeat won't even start.

`Exiting: Error reading configuration file. 1 error: Invalid top-level key 'ignore_older' found. Valid keys are bulk_queue_size, dashboards, fields, fields_under_root, logging, max_procs, name, output, path, processors, queue_size, tags, winlogbeat accessing config`

The ignore\_older setting is part of the individual event\_log config. So change it to:

```auto
winlogbeat.event_logs:
  #- name: Application
  # ignore_older: 72h
  - name: ForwardedEvents
    ignore_older: 8760h

```

There is an open issue affecting only `ForwardedEvents` and restarts. See [#3731](https://github.com/elastic/beats/issues/3731). Decreasing the `ignore_older` time after doing the first time bulk load can help alleviate the problem, but a more effective workaround is provided in that ticket I mentioned.

---

<div class="post-metadata">

**Author:** ![ssiws](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@ssiws](https://discuss.elastic.co/u/ssiws)\
**Post date:** [March 31, 2017, 9:31am UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/3 "2017-03-31T09:31:19Z")

</div>

in fact it is a formatting error when I posted the configuration on this forum, 2 spaces was deleted inadvertently.

I tried to configure logtash as mentionned on the other thread, but the duplicates are still there, here is the config file used in logstash:

```
input {
  beats {
    port => 5044
  }
}

filter {
  fingerprint {
	method => "SHA1"
	key => "_id"
	source => ["@timestamp", "computer_name","log_name","record_number"]
	concatenate_sources => true
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 31, 2017, 2:12pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/4 "2017-03-31T14:12:57Z")

</div>

In the elasticsearch output config for Logstash you have to tell it what value to use for the `_id` using [`document_id`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id). You might also want to write the generated ID into `[@metadata][id]` so that it's not in the main document.

---

<div class="post-metadata">

**Author:** ![ssiws](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@ssiws](https://discuss.elastic.co/u/ssiws)\
**Post date:** [April 3, 2017, 4:31pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/5 "2017-04-03T16:31:45Z")

</div>

Thank you for your help, it works!

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [April 4, 2017, 12:36pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/6 "2017-04-04T12:36:43Z")

</div>

@ssiws Could you please post the working LS config. I think others will find it useful.

---

<div class="post-metadata">

**Author:** ![ssiws](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@ssiws](https://discuss.elastic.co/u/ssiws)\
**Post date:** [April 4, 2017, 3:06pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/7 "2017-04-04T15:06:42Z")

</div>

Of course:

```
input {
  beats {
    port => 5044
  }
}

filter {
  fingerprint {
	method => "SHA1"
	key => "_id"
	source => ["@timestamp", "computer_name","log_name","record_number"]
	concatenate_sources => true
	target => "[@metadata][id]"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
	document_id =>"%{[@metadata][id]}"
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 2, 2017, 3:06pm UTC](https://discuss.elastic.co/t/winlogbeat-5-3-0-sends-the-whole-content-of-forwardedevents-at-startup/80596/8 "2017-05-02T15:06:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.