Winlogbeat speeding up clocks on windows servers by 1 second every minute

Elastic Stack Beats
1

Today I started to run winlogbeat on a number of Windows 2008 R2 servers and was succesfully transferring windows eventlogs to my elk stack without any problems.

We ran into a problem later on in the day with time across our domain and I'd found that one of our DCs was 5 minutes out. Monitoring the time of this server compared to the NTP servers I noticed that it was it was getting a second faster every minute. The only thing I'd changed today was to install the winlogbeat as a service. Once I stopped the service the time stopping getting increasingly ahead of the real time. I reset the clocks and monitored without winlogbeat running and the time stayed within a 2 second discepency. As soon as I started winlogbeat the time again became an extra second out after every minute. The reason my server was 5 minutes (300 seconds) out was because I'd started winbeatlog about approximately 5 hours previously (300 minutes at 1 second per minute = 300 seconds.).

Anyone else experienced a similar issue?

2

This only appears to happen if I run winlogbeat on a domain controller. If I monitor the time for 6 minutes it gets to 6 seconds out and just keeps going 1s faster per minute.

3

I can't think of anything that Winlogbeat does that would affect time. I recall hearing about a similar issue a while back related to running Filebeat (Filebeat Windows | Time sync).

4

The issue exists with 6.2.4, 6.2.3 and 6.2.0

It so far doesn't seem to exist with 6.0.0 of winlogbeat, so slowly trying to work out when it was introduced.

5

It looks like the issue came in in version 6.1.0. Problem is there is a significant fix in that version that I'd want -

Fix the registry file. It was not correctly storing event log names, and upon restart it would begin reading at the start of each event log.

6

One thing that changed between 6.0 and 6.1 was the version of Go used to compile the Beat; we switched from 1.8. to 1.9. Go 1.9 is used on the 6.1, 6.2, and 6.3 branches.

We upgraded to Go 1.10 in master. I'd be curious to know if you see this issue with a binary that was compiled using Go 1.10. Do you want to try one of the snapshot builds that are created from master? They are here.

7

Similar issue with Go 1.10, if anything it may even be quicker

image gained 1.6seconds in 1 minute

8

Sounds related.

9

Yes it does. So no chance of me using winlogbeat anytime soon then, as this issues seems to have gone quiet.

10

You can pretty easily build v6.2.4 with Go 1.8.7. There's just one patch needed to the code because it used a go1.9 feature.

diff --git a/libbeat/logp/logger.go b/libbeat/logp/logger.go
index 04cfc91a0..d7625f92f 100644
--- a/libbeat/logp/logger.go
+++ b/libbeat/logp/logger.go
@@ -4,9 +4,6 @@ import (
        "go.uber.org/zap"
 )
 
-// LogOption configures a Logger.
-type LogOption = zap.Option
-
 // Logger logs messages to the configured output.
 type Logger struct {
        sugar *zap.SugaredLogger
@@ -15,7 +12,7 @@ type Logger struct {
 // NewLogger returns a new Logger labeled with the name of the selector. This
 // should never be used from any global contexts (instead create "per instance"
 // loggers).
-func NewLogger(selector string, options ...LogOption) *Logger {
+func NewLogger(selector string, options ...zap.Option) *Logger {
        log := loadLogger().rootLogger.
                WithOptions(zap.AddCallerSkip(1)).
                WithOptions(options...).

Here's the resulting binary if you want to test it: https://s3.amazonaws.com/beats-files/testing-binaries/discuss-wlb-time-drift/winlogbeat.exe

1 Like
11

Hi Andrew

That version works without any clock drift, so I think you've found the problem.

How can I go about getting a proper released version of this, or is that unlikely to happen?

12

Can you please open a new issue in Github for this.

I don't think it's likely that there will be another 6.x release made with Go 1.8.x. I could build your the zip packages based on 6.2.4 using Go 1.8.7 if you like.

13

I confirm this is an issue on 6.2.4
I don't know if its fixed on 6.3 please reply to me if there is any fix to this issue

14

hi,
thks for Sharing this binary file but i want to build my own please guide me to setup a docker developement for building the dilebeat for example
best regard,

15

See the Contribution Guide for information on how to build beats.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.