Metricbeat writes data randomly, instead every 10 sec

[Razby (Tadej)]

Hi,

We got a problem.
Metricbeat writes data randomly even though the period=10s in case the query instance is set to all "*". The data are recorded randomly for specific instance, but not every 10 seconds.
If I set the query instance to a fixed instance e.g. "_Total" data comes exactly every 10 sec.
Tested and noticed problems for metricbeat versions 7.4.x, 7.8.x, 7.9.x.
However, it works until version 7.3.2.

The servers are not overloaded.
We have the same problem on all ElasticSearch environments where is metricbeat version higher than 7.3.2.

- module: windows
  metricsets:
    - perfmon
  period: 10s
  perfmon.ignore_non_existent_counters: true    
  perfmon.counters:
    - instance_label: logical_disk.name
      measurement_label: logical_disk.Disc.bytes.per_sec
      query: '\LogicalDisk(*)\Disk Bytes/sec' 

Thanks!

Regards,
Tadej

[MarianaD]

hi @Razby, it looks like sometimes the event is not returned. Can you set the logging.level: debug in the metricbeat.yml file and run metricbeat again.
Can you check for any exceptions and provide them to us?

[Razby (Tadej)]

Hi,

I set "logging.level: debug" and started the service again. There are no errors or warnings in the log, except those below which are expected, but data still writes randomly instead every 10 sec.

DEBUG	[perfmon]	perfmon/data.go:47	Ignoring the first measurement because the data isn't ready	{"error": "The data is not valid.", "perfmon": {"query": "\\\\TORX\\LogicalDisk(C:)\\Disk Bytes/sec"}}
WARN	[cfgwarn]	perfmon/config.go:99	DEPRECATED: perfmon.counters configuration option is deprecated and will be removed in the future major version, we advise using the perfmon.queries configuration option instead. Will be removed in version: 8.0

Regards,
Tadej

[Razby (Tadej)]

Does really nobody else has a problem with randomly data writing instead predefined interval (period ) e.g. each 10 sec?

I have this problem on all environments whether it is a newly created environment or an old existing one.

[MarianaD]

hi @Razby, if there were no errors in this case, can you check also in the logs if the missing events are indeed sent to es or they are not published at all?

[Razby (Tadej)]

Hi @MarianaD

I switched metricbeat.exe to the 32-bit version and it works.
Apparently there is a bug in the 64-bit version.
Do you already know for that bug? Will you be able to check this out?

Regards,
@razby

[MarianaD]

hi @Razby, we had some issues with the different versions, have you tested with 7.9.1 with the 64 bit version? We recently fixed an issue related to that.
If you are encountering this on 7.9.1 can you still let us if the missing events are still being published (should be found in the logs)

[Razby (Tadej)]

Hi @MarianaD

I also tried with the version 7.9.1 (64-bit) and it also doesn't work. Events are still missing.
I checked the logs but missing events were not published. No publish event exists in logs for missing events.

[MarianaD]

hi @Razby, thanks for testing this out, can you provide us with the os version you are currently running , I have been testing on Windows 10 and Windows Server 2019 and I am not able atm to reproduce the scenario.

[Razby (Tadej)]

Hi @MarianaD
I have been testing on 64-bit OS Windows 10 and Win Server 2016 DataCenter.
The problems are perfmon counters with all instances "*". There is no problem if i use static instance e.g. "_Total"

[MarianaD]

hi @Razby, can you set perfmon.ignore_non_existent_counters to false and again enable debug logging in the metricbeat.yml file and provide us with the logs again, we do expect to see errors if the expand path win 32 api has failed.

[Razby (Tadej)]

OK, I will try this.

[Razby (Tadej)]

Hi @MarianaD
I set perfmon.ignore_non_existent_counters to false and enabled debug logging.

Here are the logs that are between the intervals where the events were lost MetricBeat_logs