Credentials' variables blocking beats from running as services on Windows

[jsu]

Hey there,

So, the problem doesn't appear on linux boxes, but only windows ones : win server 2012 and win10. I had the (almost) exact same problem with Winlogbeat, Metricbeat and Filebeat.

Following the documentation concerning how to configure the output with security activated and how to do it securely with a keystore, I added the following lines to the config file:

output.elasticsearch:
  hosts: ["myEShost:9200"]
  username: "${ES_USER}"
  password: "${ES_PWD}" 

Then I actually store those variables in the keystore of the beat.

Trying to Start-Service [beat_name] doesn't work, it gives the following error:

Windows could not start the winlogbeat service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

But if I try to run the beats in the foreground with winlogbeat -e -d "*", there's no error and everything runs fine for winlogbeat and filebeat. Metricbeat also seems to run fine with no error, but it stops sending data after a few minutes (still, without any error and seeming to run fine).

If I replace the variables in the config file by the actual usernames and passwords, like this:

output.elasticsearch:
  hosts: ["myEShost:9200"]
  username: "actual_username"
  password: "actual_password" 

I can run all the Beats as Services and there's no problem at all.

Am I missing something in the documentation ? Or am I doing something else wrong ?

Looking forward to hearing from you as I really don't know where to go from here.

Thanks in advance !

Cheers,

jsu

[jkersey (Josh)]

I also am encountering this exact same issue. It works fine using the test command when the password is in the keystore, but it won't start the service if I use the keystore. it starts the service if I have the password in the yml file.

[ilovesteak]

I'm having the same issue. Extending the timeout for windows services to start doesn't fix it either.

[jkersey (Josh)]

So this actually did come down to permissions for me. It was failing to start the service when I attempted to use my normal account with elevated permissions to local administrative.

I was able to use my domain administrator account instead and this took care of it.

I had to open command prompt, use "runas /user:domain.admin@domain.com services.msc" and I was able to successfully start the service. After reboot and logon, the service status returned to "running"

Good luck to everyone else

[jsu]

Interesting, thanks ! But were you able to run the beats in the foreground before or was it also not working?

I'm also not sure about running beats as domain admin, wouldn't it be a security issue?

[jkersey (Josh)]

yes, running beats in the foreground worked, which is why it seemed weird to me that I wasn't able to simply start the service. I did get some help from Elastic (thanks Jason) and was encouraged to check permissions. Running it in the foreground didn't really show me any significant errors that I noticed, and I know that sometimes elevated rights aren't the same as running it as a different user. It seemed like the easiest thing I could try for something that should be straightforward.

I do have questions about running it as my domain admin account, and we'll probably try to set it up using a special "service" account that has the necessary permissions (not sure what those are, so there's still more work to be done). It'll kind of make sense that you don't really want your personal account running it on every computer anyway. furthermore, when you look at the properties for the service, it shows "logon as:" set to "local System account"

[jsu]

Thanks a lot for your feedback.

The idea of setting up a special service is very interesting. As you said, the main point issue will be to find what are the minimum permissions necessary for the beats to work.

[jkersey (Josh)]

Elastic support suggested that I check the debug logs to see what it didn't have permissions to access. You might need to set the debugging to a different level, such as "debug" in order to really find out.

[ferengie (Frank)]

I also have this subject.
The command test says - test ok, in an admin shell winlogbeat starts with this configuration, but as a service it doesn't work. I tried to start the service with my credentials - it doesn't work.
Unfortunately the log does not contain anything useful, it seems to be untouched. Maybe there is a problem with different path variables, because the service is configured with explicit paths and in the winlogbeat.yml there are also the paths
what information is drawn?

[jsu]

Well, this problem might need a topic of its own. It does seem to be a different problem that we had/have here.

Indeed, starting the service with hard-coded credentials in the config file works fine here. The problem appears when:

• using variables for credentials
• starting the beat as a service

And for the moment, the best solution was provided by @jkersey:

I had to open command prompt, use "runas /user:domain.admin@domain.com services.msc" and I was able to successfully start the service. After reboot and logon, the service status returned to "running"

[MarianaD]

@jsu, are you able to list the keys when defined in the keystore?
run for example:

 winlogbeat keystore list 

You might be encountering the scenario described here: https://github.com/elastic/beats/issues/12315, a workaround for this issue is found here https://github.com/elastic/beats/issues/12315#issuecomment-497777574.

[jsu]

@MarianaD I can't thank you enough, you were completely right. I was encountering this exact scenario and managed to get it fixed thanks to your links. Thank you so much!

So, for information for people who don't want to check the Github links: this is a problem in where the Beats look for the keystore. Indeed, the Beats look for data in C:\ProgramData\xxxbeat and the keystore is created by default in C:\Program Files\xxxbeat\data. Solution is either to copy the keystore in the right location (ProgramData), either to create it directly there.