Cannot start Filebeat as Windows Service

Elastic Stack Beats
1

Hello all,
I’m using Filebeat 6.0.0 on several Windows servers running Windows Server 2012 R2 Standard to send logs to Logstash. They work great!
However, last week we had IT issues on some of the servers (unexpected shutdowns, etc.). After fixing these issues, on two of the servers we haven’t been able to restore regular Filebeat behavior by running them as Windows services — when running them on CLI they work fine.
We get various error messages when trying to start them:
“Error 1067: The process terminates unexpectly.”
“Error 109: The pipe has been ended.”
“Error 1053: The service did not respond in a timely fashion.”
I tried the following steps to no avail:

  • Upgrading Filebeat to 6.3.2 on the problematic servers.
  • Increasing the timeout for starting the service in Windows Registry.
  • Restarting the servers.
  • Removing and reinstalling Filebeat.
  • Starting Filebeat using NSSM instead of services.msc. It starts the service but then it immediately fails.
  • Inspecting logs in Event Viewer. When starting using NSSM the event reads: “The Filebeat service terminated unexpectedly. It has done this X times.”
  • Adding recovery responses in services.msc in case of service fail.
  • Validating configuration — was successful.
  • Running Filebeat in the foreground — succeeds.
    Can anyone help me solve this issue?
    Thank you so much!
2

It's possible that your registry file, where filebeat stores it's internal state, has been corrupted as a result of a power failure. This happens only when run as a service because it will be using a different registry file when run from command-line.

To confirm, I would like to have a look at your logs, which also change location when run as a service. They should be in C:\programdata\filebeat\logs\filebeat.

2 Likes
3

Thank you so much for your help! Your response helped me solve my issue.
Every filebeatX.log in C:\ProgramData\filebeat\logs (I suppose each one is generated for each start attempt) indicated a registry-related failure. Here’s an excerpt from these logs:
INFO instance/beat.go:315 Filebeat start running. ... INFO registrar/registrar.go:117 Loading registrar data from C:\ProgramData\filebeat\registry ... INFO instance/beat.go:321 Filebeat stopped. ERROR instance/beat.go:691 Exiting: Could not start registrar: Error loading state: Error decoding states: invalid character '\x00' looking for beginning of value
So to solve the problem I deleted the entire C:\ProgramData\filebeat directory, although perhaps deleting the registry and the registry.old files would be enough.
Then I started the filebeat service from services.msc and now on both servers Filebeat works like a charm!
Thank you so much Adrian!

3 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.