Hi all,
We are running Metricbeat 7.6.1 on Windows 10 1909 and we notice that sometimes (once in 5 times) metricbeat does not start. Note: Metricbeat is started via Windows Service.
This is what happens in the logs:
2020-04-09T15:04:14.522+0200 INFO instance/beat.go:622 Home path: [C:\tools\metricbeat] Config path: [C:\tools\metricbeat] Data path: [C:\tools\metricbeat\data] Logs path: [C:\tools\metricbeat\logs]*
*2020-04-09T15:04:14.531+0200 INFO instance/beat.go:630 Beat ID: 9a8a098b-c0e2-40ae-affd-581c6e0cc68c*
*2020-04-09T15:04:14.531+0200 INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "C:\\tools\\metricbeat", "data": "C:\\tools\\metricbeat\\data", "home": "C:\\tools\\metricbeat", "logs": "C:\\tools\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "9a8a098b-c0e2-40ae-affd-581c6e0cc68c"}}}*
*2020-04-09T15:04:14.532+0200 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "c1c49432bdc53563e63e9d684ca3e9843626e448", "libbeat": "7.6.1", "time": "2020-02-28T23:15:46.000Z", "version": "7.6.1"}}}*
*2020-04-09T15:04:14.532+0200 INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.13.8"}}}*
*2020-04-09T15:04:14.540+0200 INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-09T12:45:24.36+02:00","name":"DESK-JORI","ip":["192.168.0.50/24","fe80::fcc2:f0bb:e071:1188/64","10.95.252.26/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.418 (WinBuild.160101.0800)","mac":["00:24:9b:4b:21:34","b8:ca:3a:7a:1b:7d"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.418"},"timezone":"CEST","timezone_offset_sec":7200,"id":"473f1c02-4853-42c8-8b67-e51e22b36468"}}}*
*2020-04-09T15:04:14.542+0200 INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\tools\\metricbeat\\metricbeat-7.6.1-windows-x86_64.exe", "name": "metricbeat-7.6.1-windows-x86_64.exe", "pid": 5064, "ppid": 768, "start_time": "2020-04-09T15:04:13.606+0200"}}}*
*2020-04-09T15:04:14.542+0200 INFO instance/beat.go:298 Setup Beat: metricbeat; Version: 7.6.1*
*2020-04-09T15:04:14.543+0200 INFO [publisher] pipeline/module.go:110 Beat name: DESK-JORI*
*2020-04-09T15:04:14.545+0200 INFO helper/privileges_windows.go:79 Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":8,"User":{"SID":"S-1-5-18","Account":"SYSTEM","Domain":"NT AUTHORITY","Type":1},"ProcessPrivs":{"SeAssignPrimaryTokenPrivilege":{"enabled":false},"SeAuditPrivilege":{"enabled_by_default":true,"enabled":true},"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePermanentPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateSymbolicLinkPrivilege":{"enabled_by_default":true,"enabled":true},"SeDebugPrivilege":{"enabled_by_default":true,"enabled":true},"SeDelegateSessionUserImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled_by_default":true,"enabled":true},"SeLoadDriverPrivilege":{"enabled":false},"SeLockMemoryPrivilege":{"enabled_by_default":true,"enabled":true},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled_by_default":true,"enabled":true},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled_by_default":true,"enabled":true},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTcbPrivilege":{"enabled_by_default":true,"enabled":true},"SeTimeZonePrivilege":{"enabled_by_default":true,"enabled":true},"SeUndockPrivilege":{"enabled":false}}}*
*2020-04-09T15:04:14.545+0200 INFO helper/privileges_windows.go:87 SeDebugPrivilege is enabled. SeDebugPrivilege=(Default, Enabled)*
*2020-04-09T15:04:14.545+0200 WARN [cfgwarn] perfmon/perfmon.go:60 BETA: The perfmon metricset is beta*
*2020-04-09T15:04:14.702+0200 INFO instance/beat.go:412 metricbeat stopped.*
*2020-04-09T15:04:14.725+0200 ERROR instance/beat.go:933 Exiting: 1 error: 1 error: initialization of reader failed: failed to expand counter (query="\Processor Information(*)\% of Maximum Frequency")
Notice the error message, we are using perfmon counters as input, but sometimes we see similar messages like this after starting Metricbeat. Also the error is not always the same perfmon counter, seems to be random.
beat.yml config:
metricbeat.modules:
- module: windows
metricsets: [perfmon]
period: 10s
perfmon.ignore_non_existent_counters: false
perfmon.counters:
# global
- instance_label: system_uptime
instance_name: uptime
measurement_label: system_uptime_seconds
query: '\System\System Up Time'
- instance_label: system_processes
instance_name: processes
measurement_label: system_processes_count
query: '\System\Processes'
- instance_label: system_threads
instance_name: threads
measurement_label: system_threads_count
query: '\System\Threads'
- instance_label: system_bytes_received
instance_name: network
measurement_label: system_network_bytes_received
query: '\Network Adapter(*)\Bytes Received/sec'
- instance_label: system_bytes_sent
instance_name: network
measurement_label: system_network_bytes_sent
query: '\Network Adapter(*)\Bytes Sent/sec'
- instance_label: system_avg_disk_queue
instance_name: disk
measurement_label: system_disk_avg_disk_queue
query: '\LogicalDisk(*)\Avg. Disk Queue Length'
- instance_label: system_curr_disk_queue
instance_name: disk
measurement_label: system_disk_curr_disk_queue
query: '\Processor(*)\% Interrupt Time'
- instance_label: system_interrupt_sec
instance_name: processor
measurement_label: system_processor_interrupt_sec
query: '\Processor(*)\Interrupts/sec'
- instance_label: system_freq_pct
instance_name: processor
measurement_label: system_processor_freq_pct
query: '\Processor Information(*)\% of Maximum Frequency'
- instance_label: system_freq
instance_name: processor
measurement_label: system_processor_freq
query: '\Processor Information(*)\Processor Frequency'
# CPU and MEM per process
- instance_label: processor.name
instance_name: total
measurement_label: processor.time.total.pct
query: '\Processor Information(_Total)\% Processor Time'
- instance_label: memory.name
instance_name: total
measurement_label: memory.bytes.private
query: '\Process(_Total)\Private Bytes'
- instance_label: processor.name
instance_name: metricbeat
measurement_label: processor.time.total.pct
query: '\Process(metricbeat)\% Processor Time'
- instance_label: memory.name
instance_name: metricbeat
measurement_label: memory.bytes.private
query: '\Process(metricbeat)\Private Bytes'
- instance_label: processor.name
instance_name: filebeat
measurement_label: processor.time.total.pct
query: '\Process(filebeat)\% Processor Time'
- instance_label: memory.name
instance_name: filebeat
measurement_label: memory.bytes.private
query: '\Process(filebeat)\Private Bytes'
fields:
beat_build: '0.0.1'
fields_under_root: true
output.kafka:
hosts:
- 1.0.0.1:9092
- 1.0.0.2:9092
- 1.0.0.3:9092
version: "0.10.2.1"
topic: 'metricbeat'
compression: gzip
compression_level: 9
monitoring:
enabled: true
cluster_uuid: '1234'
elasticsearch:
hosts:
- http://1.0.0.1:9200
- http://1.0.0.2:9200
- http://1.0.0.3:9200
Note: IP addresses are dummy addresses
I've upgraded to 7.6.1 to solve this problem, the version with the same problem was 7.4.0.
Note: The system locale is set to 'English (United States)'.
Anyone an idea what is wrong?
Thanks!
Best regards,
Jori