I'm trying to deploy metricbeat to help monitor a couple troublesome servers to see if we can pinpoint the issue. I however keep getting the following when running test modules:
error... ERROR timeout waiting for an event
Attempted on 2 2016 servers as well as a Win10 workstation to make sure I wasn't missing something. even increased the delay to 30m and it didn't change a thing.
Increasing debug just shows a bunch of access denied errors but the application is getting sedebugpriv.
Any ideas?
Can you share the output of test modules as well as the debug output?
Thanks for the response. This is just a "default" install of v6.3. Running as admin, etc etc.
C:\Temp\metricbeat-6.3.1-windows-x86_64>metricbeat.exe -c metricbeat.yml test modules -d "*" -v -e
2018-07-12T17:39:44.049-0400 INFO instance/beat.go:492 Home path: [C:\Temp\metricbeat-6.3.1-windows-x86_64] Config path: [C:\Temp\metricbeat-6.3.1-windows-x86_64] Data path: [C:\Temp\metricbeat-6.3.1-windows-x86_64\data] Logs path: [C:\Temp\metricbeat-6.3.1-windows-x86_64\logs]
2018-07-12T17:39:44.056-0400 DEBUG [beat] instance/beat.go:519 Beat metadata path: C:\Temp\metricbeat-6.3.1-windows-x86_64\data\meta.json
2018-07-12T17:39:44.064-0400 INFO instance/beat.go:499 Beat UUID: 987c873c-d9d1-4e30-ad71-daded2ed94c3
2018-07-12T17:39:44.066-0400 DEBUG [modules] beater/metricbeat.go:81 Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system, uwsgi, windows], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, etcd/leader, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/status, munin/node, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/connection, rabbitmq/node, rabbitmq/queue, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/memory, system/network, system/process, system/process_summary, system/raid, system/uptime, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, windows/perfmon, windows/service, zookeeper/mntr]]
2018-07-12T17:39:44.093-0400 DEBUG [cfgfile] cfgfile/cfgfile.go:143 Load config from file: C:\Temp\metricbeat-6.3.1-windows-x86_64\modules.d\system.yml
2018-07-12T17:39:44.115-0400 INFO helper/privileges_windows.go:62 Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":8,"User":{"SID":"S-1-5-21-3046630171-3305385774-927799177-1001","Account":"justi","Domain":"DESKTOP-8MFBE3N","Type":1},"ProcessPrivs":{"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled":false},"SeCreateSymbolicLinkPrivilege":{"enabled":false},"SeDebugPrivilege":{"enabled":false},"SeDelegateSessionUserImpersonatePrivilege":{"enabled":false},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled":false},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled":false},"SeLoadDriverPrivilege":{"enabled":false},"SeLockMemoryPrivilege":{"enabled":false},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled":false},"SeRemoteShutdownPrivilege":{"enabled":false},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled":false},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTimeZonePrivilege":{"enabled":false},"SeUndockPrivilege":{"enabled":false}}}
2018-07-12T17:39:44.125-0400 INFO helper/privileges_windows.go:94 SeDebugPrivilege is now enabled. SeDebugPrivilege=(Enabled)
system...
process...2018-07-12T17:39:45.221-0400 DEBUG [module] module/wrapper.go:154 Starting metricSetWrapper[module=system, name=process, host=]
2018-07-12T17:39:45.229-0400 DEBUG [processes] process/process.go:443 Skip process pid=0: error getting process state for pid=0: getProcName failed: OpenProcess failed for pid=0: The parameter is incorrect.; getProcStatus failed: OpenProcess failed for pid=0: The parameter is incorrect.; getParentPid failed: OpenProcess failed for pid=0: The parameter is incorrect.; getProcCredName failed: OpenProcess failed for pid=0: The parameter is incorrect.
2018-07-12T17:39:45.230-0400 DEBUG [processes] process/process.go:443 Skip process pid=4: error getting process state for pid=4: getProcName failed: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument; getProcCredName failed: OpenProcess failed for pid=4: Access is denied.
2018-07-12T17:39:45.238-0400 DEBUG [processes] process/process.go:443 Skip process pid=128: error getting process state for pid=128: getProcCredName failed: OpenProcess failed for pid=128: Access is denied.
2018-07-12T17:39:45.247-0400 DEBUG [processes] process/process.go:443 Skip process pid=804: error getting process state for pid=804: getProcCredName failed: OpenProcess failed for pid=804: Access is denied.
2018-07-12T17:39:45.249-0400 DEBUG [processes] process/process.go:443 Skip process pid=908: error getting process state for pid=908: getProcCredName failed: OpenProcess failed for pid=908: Access is denied.
2018-07-12T17:39:45.594-0400 ERROR process/process.go:454 Error getting process details. pid=752: error getting process arguments for pid=752: ProcArgs failed for pid=752: could not get Win32_Process WHERE ProcessId = 752: wmi: cannot load field "CommandLine" into a "string": unsupported type ()
error... ERROR timeout waiting for an event
now i hit ctrl+c this time around, if I don't, the error's will keep pooling in the screen, yet it's moved on to other modules.
I did pull out information related to the other metrics gathered to make the log "fit" in a reply. and only contain relevant info.
What is the user you start metricbeat with? It seems the user misses some access rights. Can you share your metricbeat config?
Also I'm not sure the error comes from the module config but from the internal stats . Could you try to set xpack.monitoring.enabled: false in your config and see if you still get the error?
In that log i'm running it as my user account which is local administrator to the workstation which is not domain joined. It is windows and UAC is enabled, so I'm running it via a command prompt that was started with right clicking and running as admin to get the UAC prompt. I've also ran it under psexec with the system and interactive flags so I was the system user and had the same issues.
xpack monitoring looks to be disabled by default as the config says to specify true to enable, I have however added the line and specifying false and receive the same error.
I've even compiled from git and have the same issue. I assume it's some sort of setting issues as I don't see a widespread report of it.
Both machines are running docker containers on windows via hyper-v, when I checked the errors for some of the PIDS they were related to lsa, vmmem, etc. So to me they seemed like high level system processes that we most likely have access to(Pid 0, pid 4)
So it looks like this only effects the module tests. When running it, it works fine. I tracked it down to a commit https://github.com/elastic/beats/pull/5835 which @ruflin you actually commented on as an issue with tests failing.
Thanks for all your help everyone but if it's just tests failing I can deal with that.
I think the tests referred there is not the same as test modules. Is the output you shared above from test modules or when you run the Beat?
You mentioned above docker containers in windows: So the Beat is running inside the docker container? Or the errors you see are for the docker containers running?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.