Agent Unable to Start Service

Hi,

I am having issues with this 1 agent (I have 26 in my environment and all is running fine except for this 1 agent).

When trying to start the service, it would start awhile and then shutdown on its own.

Tried reinstalling the agent and this is the error:

[   =] Service Started  [21s] Elastic Agent successfully installed, starting enrollment.
[  ==] Waiting For Enroll...  [23s] {"log.level":"warn","@timestamp":"2024-05-06T12:18:16.831+0800","log.logger":"tls",
log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.ve
sion":"1.6.0"}
[   =] Waiting For Enroll...  [24s] {"log.level":"info","@timestamp":"2024-05-06T12:18:17.541+0800","log.origin":{"file
name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://prod-00001.simplydata.com.my:
220/","ecs.version":"1.6.0"}
[====] Waiting For Enroll...  [24s] {"log.level":"warn","@timestamp":"2024-05-06T12:18:17.777+0800","log.logger":"tls",
log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.ve
sion":"1.6.0"}
[ ===] Waiting For Enroll...  [35s] {"log.level":"info","@timestamp":"2024-05-06T12:18:28.905+0800","log.origin":{"file
name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:18:28.908+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":496},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"'. Wa
ting for 2s","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [37s] {"log.level":"info","@timestamp":"2024-05-06T12:18:30.910+0800","log.origin":{"file
name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:18:30.911+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":496},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"'. Wa
ting for 4s","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [41s] {"log.level":"info","@timestamp":"2024-05-06T12:18:34.913+0800","log.origin":{"file
name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:18:34.914+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":496},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"'. Wa
ting for 8s","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [49s] {"log.level":"info","@timestamp":"2024-05-06T12:18:42.918+0800","log.origin":{"file
name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:18:42.919+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":496},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"'. Wa
ting for 16s","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [1m5s] {"log.level":"info","@timestamp":"2024-05-06T12:18:58.922+0800","log.origin":{"fil
.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:18:58.923+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":496},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"'. Wa
ting for 32s","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-06T12:19:30.928+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.li
e":296},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all
etries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while diali
g: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"","ecs.version":"1.6.0"}
Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error
 rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elast
c-agent-system: The system cannot find the file specified."
[ ===] Waiting For Enroll...  [2m0s] Error: could not reload agent daemon, unable to trigger restart: could not reload
gent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport
 Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
[    ] Uninstalled  [2m3s] Error uninstalling.  Printing logs
{debug 2024-05-06 12:19:54.0308324 +0800 +08 m=+122.040796701 processes Error fetching PID info for 0, skipping: GetInf
ForPid: could not get all information for PID 0: error fetching name: OpenProcess failed for pid=0: The parameter is in
orrect.
error fetching status: OpenProcess failed for pid=0: The parameter is incorrect. github.com/elastic/elastic-agent-syste
-metrics@v0.9.2/metric/system/process/process.go:173 }
{debug 2024-05-06 12:19:54.0308324 +0800 +08 m=+122.040796701 processes Error fetching PID info for 4, skipping: GetInf
ForPid: could not get all information for PID 4: error fetching name: GetProcessImageFileName failed for pid=4: GetProc
ssImageFileName failed: invalid argument github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/system/process/p
ocess.go:173 }
{debug 2024-05-06 12:19:54.0318329 +0800 +08 m=+122.041797201 processes Non fatal error fetching PID some info for 512,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: OpenProcess failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/sy
tem/process/process.go:176 }
{debug 2024-05-06 12:19:54.0318329 +0800 +08 m=+122.041797201 processes Non fatal error fetching PID some info for 656,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: OpenProcess failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/sy
tem/process/process.go:176 }
{debug 2024-05-06 12:19:54.0328418 +0800 +08 m=+122.042806001 processes Non fatal error fetching PID some info for 736,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: OpenProcess failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/sy
tem/process/process.go:176 }
{debug 2024-05-06 12:19:54.0328418 +0800 +08 m=+122.042806001 processes Non fatal error fetching PID some info for 780,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: OpenProcess failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/sy
tem/process/process.go:176 }
{debug 2024-05-06 12:19:54.0348345 +0800 +08 m=+122.044798601 processes Non fatal error fetching PID some info for 132,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0358292 +0800 +08 m=+122.045793201 processes Non fatal error fetching PID some info for 1152
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.042847 +0800 +08 m=+122.052810601 processes Non fatal error fetching PID some info for 2968,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.042847 +0800 +08 m=+122.052810601 processes Non fatal error fetching PID some info for 3032,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.042847 +0800 +08 m=+122.052810601 processes Non fatal error fetching PID some info for 2256,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.042847 +0800 +08 m=+122.052810601 processes Non fatal error fetching PID some info for 3100,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.042847 +0800 +08 m=+122.052810601 processes Non fatal error fetching PID some info for 3140,
metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to f
tch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/
etric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0438417 +0800 +08 m=+122.053805201 processes Non fatal error fetching PID some info for 3292
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
[   =] Uninstalled  [2m3s] {debug 2024-05-06 12:19:54.0438417 +0800 +08 m=+122.053805201 processes Non fatal error fetc
ing PID some info for 3324, metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args
 Not enough privileges to fetch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-
gent-system-metrics@v0.9.2/metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0438417 +0800 +08 m=+122.053805201 processes Non fatal error fetching PID some info for 3460
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0438417 +0800 +08 m=+122.053805201 processes Non fatal error fetching PID some info for 3660
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0468525 +0800 +08 m=+122.056815901 processes Non fatal error fetching PID some info for 4540
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0958757 +0800 +08 m=+122.105836301 processes Non fatal error fetching PID some info for 1291
, metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
fetch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.
/metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0958757 +0800 +08 m=+122.105836301 processes Non fatal error fetching PID some info for 9668
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0968807 +0800 +08 m=+122.106841101 processes Non fatal error fetching PID some info for 2667
, metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
fetch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.
/metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0978567 +0800 +08 m=+122.107817201 processes Non fatal error fetching PID some info for 3160
 metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
etch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2
metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.0988851 +0800 +08 m=+122.108845501 processes Non fatal error fetching PID some info for 2352
, metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
fetch information: OpenProcess failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.2/metric/
ystem/process/process.go:176 }
{debug 2024-05-06 12:19:54.1008798 +0800 +08 m=+122.110840101 processes Non fatal error fetching PID some info for 2115
, metrics are valid, but partial: FillMetricsRequiringMoreAccess: error fetching process args: Not enough privileges to
fetch information: GetUserProcessParams failed: Access is denied. github.com/elastic/elastic-agent-system-metrics@v0.9.
/metric/system/process/process.go:176 }
{debug 2024-05-06 12:19:54.1719341 +0800 +08 m=+122.181890301  Loaded configuration from C:\Users\ws341\Downloads\elast
c-agent-8.13.1-windows-x86_64\elastic-agent-8.13.1-windows-x86_64\elastic-agent.yml github.com/elastic/elastic-agent/in
ernal/pkg/config/loader.go:45 }
{debug 2024-05-06 12:19:54.1719341 +0800 +08 m=+122.181890301  Merged configuration from C:\Users\ws341\Downloads\elast
c-agent-8.13.1-windows-x86_64\elastic-agent-8.13.1-windows-x86_64\elastic-agent.yml into result github.com/elastic/elas
ic-agent/internal/pkg/config/loader.go:57 }
{debug 2024-05-06 12:19:54.1719341 +0800 +08 m=+122.181890301  Merged all configuration files from [C:\Users\ws341\Down
oads\elastic-agent-8.13.1-windows-x86_64\elastic-agent-8.13.1-windows-x86_64\elastic-agent.yml], no external input file
 github.com/elastic/elastic-agent/internal/pkg/config/loader.go:64 }
{debug 2024-05-06 12:19:54.1719341 +0800 +08 m=+122.181890301 composable Starting controller for composable inputs gith
b.com/elastic/elastic-agent/internal/pkg/composable/controller.go:118 }
{debug 2024-05-06 12:19:54.1729156 +0800 +08 m=+122.182871701 composable Started controller for composable inputs githu
.com/elastic/elastic-agent/internal/pkg/composable/controller.go:161 }
{debug 2024-05-06 12:19:54.1729156 +0800 +08 m=+122.182871701 composable Variable state changed for composable inputs;
ebounce started github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:197 }
{debug 2024-05-06 12:19:54.1729156 +0800 +08 m=+122.182871701 docker Docker client will negotiate the API version on th
 first request. github.com/elastic/elastic-agent-autodiscover@v0.6.7/docker/client.go:49 }
{info 2024-05-06 12:19:54.1729156 +0800 +08 m=+122.182871701 composable.providers.docker Docker provider skipped, unabl
 to connect: protocol not available github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker/docker.go
44 }
{debug 2024-05-06 12:19:54.1739386 +0800 +08 m=+122.183894601 composable kubernetes_secrets provider skipped, unable to
connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setti
g KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetess
crets/kubernetes_secrets.go:81 }
{debug 2024-05-06 12:19:54.1739386 +0800 +08 m=+122.183894601 composable.providers.kubernetes Kubernetes provider for r
source pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuratio
 has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/co
posable/providers/kubernetes/kubernetes.go:106 }
{debug 2024-05-06 12:19:54.1739386 +0800 +08 m=+122.183894601 composable Kubernetes leaderelection provider skipped, un
ble to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, tr
 setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kube
netesleaderelection/kubernetes_leaderelection.go:53 }
{debug 2024-05-06 12:19:54.1739386 +0800 +08 m=+122.183894601 composable.providers.kubernetes Kubernetes provider for r
source node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configurati
n has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/c
mposable/providers/kubernetes/kubernetes.go:106 }
{debug 2024-05-06 12:19:54.2730363 +0800 +08 m=+122.282986701 composable Computing new variable state for composable in
uts github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:213 }
{debug 2024-05-06 12:19:54.2730363 +0800 +08 m=+122.282986701 composable Stopping controller for composable inputs gith
b.com/elastic/elastic-agent/internal/pkg/composable/controller.go:165 }
{debug 2024-05-06 12:19:54.3750961 +0800 +08 m=+122.385040501 composable Stopped controller for composable inputs githu
.com/elastic/elastic-agent/internal/pkg/composable/controller.go:194 }
Error: enroll command failed for unknown reason: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html

From the logs, I can see its main issue is:

rpc error: code = Unavailable desc = connection error: desc = \"transport
 Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\

After researching around, it seems that this have to do with the service user. I've tried local system user and also AD administrator user, both doesn't work.

I've also tried it with version 8.9.2, 8.13.1, and 8.13.3. All doesn't work either.

FYI, this is a Windows Server 2012 machine (I know it's old) but version 8.9.x still supports it. My fleet is on 8.13.1.

Hello Eric:

I am having a similar issue on a Windows 2019 Server VM and have had a support case open since February. Have you found a solution?

Thanks,
Nate

No, haven't found the issue yet. For now, we work with log forwarding. We forward the logs to another server and capture the logs from there instead.

It's only this server which is having issue. Not sure what are the conflicting software/drivers with Windows. If there's any reply from support, do update here too. I'm sure we're not the only people having this issue.