Winlogbeat output to Graylog

[justas]

Hi All,

I have an issue. My winlogbeat sends duplicate and even in one event I found x150 same messages in graylog where in windows event logs there is only 1 message.

Winlogbeat: 5.5.2
Graylog: 2.3.0
Elasticsearch: 5.5.2

winlogbeat.event_logs:
name: ForwardedEvents
batch_read_size: 512

output.logstash: # used by Graylog
hosts: [“GraylogIP:5044”]
ssl.enabled: true
ssl.verification_mode: none
compression_level: 3
bulk_max_size: 512

There can be 100x events with the same “winlogbeat_record_number” however different graylog message code. All those 100x messages are identical. Anyone is facing the same issue?

[steffens (Steffen Siering)]

No idea about Graylog. But winlogbeat resends events on failure (missing ACK). Checks logs and metrics in logs about send failures.

[justas]

I have these messages from time to time (every 30 seconds):

2017-10-24T00:15:32+03:00 INFO Non-zero metrics in the last 30s:
libbeat.logstash.call_count.PublishEvents=30
libbeat.logstash.publish.read_bytes=1050
libbeat.logstash.publish.write_bytes=811609
libbeat.logstash.published_and_acked_events=2841
libbeat.publisher.published_events=2841
published_events.ForwardedEvents=2841
published_events.total=2841
read_errors.1734=10
uptime={"server_time":"2017-10-23T21:15:32.5831811Z",
"start_time":"2017-10-3T10:01:31.4012226Z",
"uptime":"11h14m1.1819585s",
"uptime_ms":"40441181958"}

I cannot find what that means "non-zero metrics". From what I see, all correct here.

[andrewkroh]

Do the duplicates only occur after restarting Winlogbeat?

If so you are seeing https://github.com/elastic/beats/issues/3731#issuecomment-290452499 which we plan on making some changes in a 6.x minor release to fix. But as long as you aren't restarting often the problem is minimal, and if you using a Logstash fingerprint filter you can completely work around the issue.

[steffens (Steffen Siering)]

This is no error message. The message is printed every 30 seconds, printing out only metrics that have changed since the last 30 seconds.

The published_and_acked_events metric indicates events have been successfully published. No retries or fails it seems.