Winlogbeat gives unathorized error when using keystore

[kaushas (AndyK)]

When using winlogbeat Keystore for PWD, I get 401 unauthorized error. Just passing elastic password value itself in the config file works.
I have looked at few of these issues raised by others already but none of the solution outlined there is working.
We are not using windows service yet. It is simple command line invocation. We have same issue with apm-server as well. Below are the details.

1. Created a new keystore, it was created in data folder
2. Added ES_PWD with pwd for elasticsearch
3. Check with list command and it displays the key that was entered
4. Change the config file output.elasticsearch.password : "${ES_PWD}" just as the documentation states. Does not work give 401 error
5. Verified the Config file. winlogbeat.exe test config -e , says config OK, with two warn

2020-06-25T19:36:49.462-0400    INFO    instance/beat.go:655    Beat ID: dbbfd418-dcb5-4987-8d3f-77695dbf723f
2020-06-25T19:36:49.475-0400    INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "F:\\Elastic", "data": "F:\\Elastic\\data", "home": "F:\\Elastic", "logs": "F:\\Elastic\\logs"}, "type": "winlogbeat", "uuid": "dbbfd418-dcb5-4987-8d3f-77695dbf723f"}}}
2020-06-25T19:36:49.475-0400    INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:26:46.000Z", "version": "7.8.0"}}}
2020-06-25T19:36:49.475-0400    INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-06-25T19:36:49.483-0400    INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-30T23:20:03.83-04:00","name":"WXXXXXX","ip":["10.XX.XX.XXX/24","::1/128","127.0.0.1/8","feXX::xxxx:xxx:3def/128"],"kernel_version":"10.0.14393.3686 (rs1_release.200504-1524)","mac":["00:50:33:00:00:00","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3686"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"6771528f-117e-4e75-8cc2-7bd196361373"}}}
2020-06-25T19:36:49.483-0400    INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"cwd": "F:\\Elastic", "exe": "F:\\Elastic\\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 6236, "ppid": 11888, "start_time": "2020-06-25T19:36:48.523-0400"}}}
2020-06-25T19:36:49.484-0400    INFO    instance/beat.go:310    Setup Beat: winlogbeat; Version: 7.8.0
2020-06-25T19:36:49.487-0400    INFO    [index-management]      idxmgmt/std.go:183      Set output.elasticsearch.index to 'winlogbeat-7.8.0' as ILM is enabled.
2020-06-25T19:36:49.488-0400    INFO    eslegclient/connection.go:97    elasticsearch url: http://wXXXXXX:9200
2020-06-25T19:36:49.489-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: WXXXXX
2020-06-25T19:36:49.489-0400    INFO    beater/winlogbeat.go:69 State will be read from and persisted to F:\Elastic\data\.winlogbeat.yml
2020-06-25T19:36:49.505-0400    WARN    [cfgwarn]       registered_domain/registered_domain.go:60   BETA: The registered_domain processor is beta.
2020-06-25T19:36:49.540-0400    WARN    [cfgwarn]       registered_domain/registered_domain.go:60   BETA: The registered_domain processor is beta

5. Moved the keystore to C\:programdata\winlogbeat/... keystore
6. Tried starting it does not work - 401 error
7. Tried using : winlogbeat.exe -E keystore.path="C:\ProgramData\winlogbeat\winlogbeat.keystore"

Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://wXXXXXXXXX:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [elastic] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}.```
8. Even tried "$ES_PWD" which was outlined as a solution to this problem, same results
9. I have looked at the bug on github which is prompted me to try step #7 above.

Has anyone else solve this problem? Really do not want pwd in clear text on the server. Any help/ideas will be much appreciated.

thanks

[MarianaD]

hi @kaushas, have you tried running:
winglogbeat keystore create -E keystore.path= C:\ProgramData\winlogbeat
which should create the keystore in the specified location then adding the password:
winlogbeat keystore add ES_PWD
Let us know what the output is.

[kaushas (AndyK)]

I get this error and I have another person try it (both admin access)
F:\Elastic>winlogbeat keystore create -E keystore.path=c:\programdata\winlogbeat\

error initializing beat: could not initialize the keystore: read c:\programdata\winlogbeat: The handle is invalid.

However this works
F:\Elastic>winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_PWD

The keystore does not exist. Do you want to create it? [y/N]: Y

Created keystore

Enter value for ES_PWD:

Successfully updated the keystore