Winlogbeat - unable to start service on windows

[fracorbas]

@marc.guasch

When you mention this, to which account you mean? There are Service and Computer accounts, have you tried both?

I have 3 choices :

• the certificates of the local computer (which gives me the choice to get the certificate of a computer of a domain or workgroup and the certificates of my local machine).
• the certificates of a "service user" in which I can chose for a service user like winlogbeat, but there is no System user or anything that is close to it.
• the certificates of the user i'm logged with

when I mentionned this it was in the local computer not an account, and I found nothing on internet to change the certificates of the System user.
but it seems that changing directly in the local machine changed the certificate of my user and of the service user, so I think it may have changed the certificate for every user, but I'm not sure.
I also tried here to change the certificates of the winlogbeat service user but it changed nothing

@marc.guasch

On top of that, could be interesting to try to grab some logs about why it is failing, maybe there are some more detailed ones in the Event viewer (System log)? Maybe we can get a bit more specific error message there.

I'm not in front of the server right now, I'll looked for that tomorrow, but the only messages I got with sysmon and in eventvwr was the delay, and that it could not start.

the frustrating thing here is that the .exe works, so I cannot have the log directly in the console to see where the service stop.

I'll see what I can find tomorrow.

[fracorbas]

@marc.guasch

On top of that, could be interesting to try to grab some logs about why it is failing, maybe there are some more detailed ones in the Event viewer (System log)? Maybe we can get a bit more specific error message there.

I tried looking deep in the windows system log and found absolutely nothing. To be sure I forgot nothing, I exported the log in ELK with the exe and looked for winlogbeat and the error that were happening :

so it's just absolutely not helping, and same in eventvwr I can't find an error that would tell me why it's not working.

So I looked up to the log folder in winlogbeat and found errors like this :

{"log.level":"warn","@timestamp":"2024-05-07T09:24:59.607+0200","log.logger":"winlogbeat","log.origin":{"function":"github.com/elastic/beats/v7/winlogbeat/beater.(*eventLogger).run","file.name":"beater/eventlogger.go","file.line":154},"message":"Open() encountered channel not found error. Trying again...","service.name":"winlogbeat","id":"{"log.level":"warn","@timestamp":"2024-05-07T10:06:08.904+0200","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.equalsIntValue.func1","file.name":"conditions/equals.go","file.line":37},"message":"expected int but got type string in equals condition.","service.name":"winlogbeat","ecs.version":"1.6.0"}","error":{"message":"Le canal spécifié est introuvable."},"channel":"Security_User_Lock_Out","ecs.version":"1.6.0"}

The reason is I tried to take a file on internet, but some channels do not exist on my server. So I deleted every channel that were making an error. Now those errors are not happening.
And : the service is not starting.

But I realized something : the log folder is only filled when I start the exe in console, it is never filled when I start the service, so I have no clue to see what is causing this.

I still found strange things like :

{"log.level":"warn","@timestamp":"2024-05-07T09:27:46.503+0200","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.equalsIntValue.func1","file.name":"conditions/equals.go","file.line":37},"message":"expected int but got type string in equals condition.","service.name":"winlogbeat","ecs.version":"1.6.0"}

but I dont have the line in the file or where the condition is causing this. I only found those lines :

setup.kibana.protocol: https # it was protocol: "https"

so I removed every " in the arguments but it does nothing, this line still apears.

and here's the informations I found :

{"log.level":"info","@timestamp":"2024-05-07T10:19:24.945+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":811},"message":"Home path: [C:\\Program Files\\Winlogbeat] Config path: [C:\\Program Files\\Winlogbeat] Data path: [C:\\Program Files\\Winlogbeat\\data] Logs path: [C:\\Program Files\\Winlogbeat\\logs]","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:24.999+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1374},"message":"Build info","service.name":"winlogbeat","system_info":{"build":{"commit":"26aad5d437d592cea2d8d3e0b950f885ff47fe41","libbeat":"8.13.0","time":"2024-03-22T02:41:42.000Z","version":"8.13.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.040+0200","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection","file.name":"eslegclient/connection.go","file.line":122},"message":"elasticsearch url: https://<ip>:9200","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.041+0200","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings","file.name":"pipeline/module.go","file.line":105},"message":"Beat name: LRFRESEAU02S","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.041+0200","log.logger":"winlogbeat","log.origin":{"function":"github.com/elastic/beats/v7/winlogbeat/beater.New","file.name":"beater/winlogbeat.go","file.line":70},"message":"State will be read from and persisted to C:\\Program Files\\Winlogbeat\\data\\.winlogbeat.yml","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.043+0200","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry","file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"winlogbeat","input_type":"winlog","id":"OpenSSH/Operational","key":"OpenSSH/Operational","uuid":"578c326b-e9ff-40da-a03d-60c7081e0577","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.062+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch","file.name":"instance/beat.go","file.line":520},"message":"winlogbeat start running.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-07T10:19:25.062+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop","file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-05-07T10:19:25.326+0200","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.equalsIntValue.func1","file.name":"conditions/equals.go","file.line":37},"message":"expected int but got type string in equals condition.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-05-07T10:19:25.326+0200","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.equalsIntValue.func1","file.name":"conditions/equals.go","file.line":37},"message":"expected int but got type string in equals condition.","service.name":"winlogbeat","ecs.version":"1.6.0"}

[fracorbas]

the system log I found are :

• 

I did a ctrl + f to look for winlogbeat and the only log that exists are those errors (the delay...)

[fracorbas]

@marc.guasch

Hello!, since the issue seems to only appear when https is enabled, do you know if your certificate is trusted system wide so it can be used by services?

and I forgot something : generating the private key and importing it in my windows, I used this command :

openssl pkcs12 -export -out <domain name>.pfx -inkey <domain name>.key -in <domain name>crt

imported it in windows :

The key is recognized, but the service is still not starting.

[fracorbas]

@marc.guasch

Is there a chance you can test this in another environment to confirm it is the issue?

I just tried with another computer, just to see if the service would start.

I did the same as on my server, installing every certificate, installing the service, and then launching it.

I got the same error that the service cannot start.
I recieve the same log :

{
    "log.level": "warn",
    "@timestamp": "2024-05-07T14:32:15.674+0200",
    "log.logger": "conditions",
    "log.origin": {
        "function": "github.com/elastic/beats/v7/libbeat/conditions.equalsIntValue.func1",
        "file.name": "conditions/equals.go",
        "file.line": 37
    },
    "message": "expected int but got type string in equals condition.",
    "service.name": "winlogbeat",
    "ecs.version": "1.6.0"
}

which I still dont understand, because there is no string value in my file...

by the way, the link in the logs is not the right one, I think the github tree structure has changed, it should be : beats/libbeat/conditions at main · elastic/beats · GitHub

I'm kinda stuck right now, I hope I have not missed something really simple that causes this.

[s0p4L1n3]

Can you try this config ?

setup.kibana:
  host: "https://<server_ip>:5601"
  #protocol: "https"
  #ssl.certificate_authorities: ["certs/ca.crt"]
  ssl.enabled: true
  username: "elastic"
  password: "<my_password>"

output.elasticsearch:
  hosts: ["https://<server_IP>:9200"]
  username: "elastic"
  password: "<my_password>"
  #ssl.certificate_authorities: ["certs/ca.crt"]
  ssl.enabled: true
  pipeline: "winlogbeat-%{[agent.version]}-routing"

• As the CA is correctly installed in the trust store, you don't need to specify the ssl.certitifcate_authorities, it will automatically look for it.
• You don't need to specifies protocol: https because you already declare https:// in hosts: for elasticsearch output
• Be careful with setup.kibana, you need to declare host: not hosts:
• Also in the reference yml, host value is not between brackets.

It can be explained because output.elasticsearch is looking for an array of hosts while kibana.setup is looking only for one host.

I can confirm because I've just added the brackets and the s to my setup.kibana settings and I got this error:

Try it and check if it changes the behaviour ?

winlogbeat.exe test conf check just if your YAML file does not contains indentation erros etc... it does not verify between hosts and host.

[fracorbas]

I just tried what you said, and the service starts !

I don't really know why the errors in the logs are not really useful to troubleshoot what is happening. I still don't know which line was generating this issue, but now it's no longer here.

Thank you very much for this.

Have a great day

[s0p4L1n3]

I still think the problem was your line:

setup.kibana:
  hosts: "https://<server_ip>:5601"

and it should be

setup.kibana:
  host: "https://<server_ip>:5601"

Because winlogbeat is looking for a single server and not an array of servers, that's why it need host instead of hosts

If you add again the s, and try to restart winlogbeat it will fail.

Good luck with your journey on ELK !