Issue with Keystore on winlogbeat

sbemiller · December 29, 2021, 3:03pm

Elastic 7.15.1

I have read through a lot of discussions on how to solve this issue but nothing is working.
I am working on an install process for the winlogbeat service.
The Service installs and starts with no issues showing.
But if you go the logs and review, I get the following error messages:

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(Elasticsearch(https://IP_ADDRESS:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [winlogbeat-input] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [winlogbeat-input] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}},"status":401}

INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(Elasticsearch(https://IP_ADDRESS:9200)) with 1 reconnect attempt(s)

INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer

INFO [publisher] pipeline/retry.go:223 done

I also recreated the Keystore, using the following commands to see if would resolve the issue ;
winlogbeat keystore create -E keystore.path= C:\ProgramData\winlogbeat
winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add DFWES_PWD.

If I remove the Keystore Password within the winlogbeat.yml file, it works just fine and connects.

The .yml file shows this for the User SetUp:
username: "winlogbeat-input"
password: "${DFWES_PWD}"

Does anyone have any hints, or ideas on how to do this using the KeyStore ?

HINT on the issue;
I also made sure that the Keystore is in the ProgramData folder. I did a copy command on the Keystore file. When I "MOVE" the Keystore ( from Program Files to ProgramData ) the Service is not finding the file. That means the service is looking at Program Files for the Keystore.

How do I change this to look at "ProgramData",
I have tried the folowing command but it is not working
winlogbeat.exe -e keystore.path = "C:\ProgramData\winlogbeat\winlogbeat.keystore"

I also recreated the Keystore to make sure it was in ProgramData\winlogbeat
winlogbeat keystore create -E keystore.path= C:\ProgramData\winlogbeat
and added the password for the Key and verified that it was there.

stephenb · December 29, 2021, 4:27pm

3 Things I would check...

The keystore file permissions.... make sure it is readable and the correct user that winlogbeat service is running as.

The keystore command must be run by the same user who will run Winlogbeat.

Also did you do a list to see if the DFWES_PWD is in the keystore.

winlogbeat keystore list

Also I am sure you tried this ... but did you try without all the paths... simply... on the other systems when you do not specify the paths... it figures out all the correct paths by itself.

winlogbeat keystore create
winlogbeat keystore add DFWES_PWD
winlogbeat keystore list

Also I assume you installed exactly has shown here

andrewkroh · December 29, 2021, 4:45pm

The keystore is located by looking in path.data. (ref). The windows service sets path.data to programdata. You can look at the properties of the service to verify this. So I think rather than setting keystore.path you should set -E path.data=C:/ProgramData/winlogbeat when manually running the exe (mimic what the service's command does). Also note the -E vs -e.

andrewkroh · December 29, 2021, 4:48pm

Also for debugging I would try to cause an error when the variable is not found via the keystore. See Use environment variables in the configuration | Winlogbeat Reference [7.16] | Elastic. This way you get a hard failure long before it tries to connect to ES.

While this is about env vars I think it applies to keystore vars too. So try ${FOO:?FOO not found in keystore} as the format in your config file.

sbemiller · December 29, 2021, 5:24pm

I went in and checked the file permissions on the Keystore and they are at read and execute.
I also made sure on the correct user.

Just to be sure on the Keystore commands that you have listed. I reran them all and tried it again, same error message.

sbemiller · December 29, 2021, 5:38pm

I checked the path on the "service";

"C:\Program Files\winlogbeat\winlogbeat.exe" --environment=windows_service -c "C:\Program Files\winlogbeat\winlogbeat.yml" --path.home "C:\Program Files\winlogbeat" --path.data "C:\ProgramData\winlogbeat" --path.logs "C:\ProgramData\winlogbeat\logs" -E logging.files.redirect_stderr=true

Manually, I also executed winlogbeat -E and the path.data as shown in the post. Nothing appears in the Command box. It just sits until I do a Ctrl-C to stop it.

I will be trying the debugging next to see if that helps on the issue.

sbemiller · December 29, 2021, 7:38pm

Weird, I placed this in the config file ;
password: "${DFWES_PWD:?DFWES_PWD not found in keystore}"

No error messaged appeared on the password not being found within the Keystore.

I am still getting the same Error Message BUT when I enter in the "password" in the config file, it connects and sends the data. The following is some of the information that I receive when I manually start the service using the Keystore. ( removed the IP address of the host(s) )

QUESTION : Does the Keystore have issues with Special Characters, the only characters being used are |, @ and +

2021-12-29T11:18:49.832-0800 2021-12-29T11:18:49.832-0800 2021-12-29T11:18:52.819-0800 2021-12-29T11:18:53.823-0800 2021-12-29T11:18:53.823-0800 2021-12-29T11:18:53.823-0800 2021-12-29T11:18:53.824-0800 2021-12-29T11:18:53.825-0800 2021-12-29T11:18:53.825-0800 2021-12-29T11:18:53.825-0800 2021-12-29T11:18:53.825-0800 2021-12-29T11:18:53.826-0800 2021-12-29T11:18:53.826-0800 2021-12-29T11:18:53.826-0800 2021-12-29T11:18:53.826-0800 2021-12-29T11:18:53.827-0800 2021-12-29T11:18:53.827-0800 2021-12-29T11:18:53.827-0800 2021-12-29T11:18:53.827-0800 2021-12-29T11:18:55.047-0800 INFO instance/beat.go:473 winlogbeat start running.
INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(https://IP_Address:9200))
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
INFO [publisher] pipeline/retry.go:223 done
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(https://IP_Address:9200))
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
INFO [publisher] pipeline/retry.go:223 done
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(https://IP_Address:9200))
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
INFO [publisher] pipeline/retry.go:223 done
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(https://IP_Address:9200))
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
INFO [publisher] pipeline/retry.go:223 done
ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(Elasticsearch(https://IP_Address:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [winlogbeat-input] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [winlogbeat-input] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}},"status":401}

andrewkroh · December 29, 2021, 7:55pm

Try running with -d config-with-passwords. That should log the config YAML with passwords shown. Hopefully it’s after the keystone lookups. (On PTO from mobile so can’t test now.)

github.com

elastic/beats/blob/3598bd8d5bcf61e31cd7d81b8d2af4611b324b4a/libbeat/common/config_test.go#L104

    
      
              "headers": {
                "Authorization": "xxxxx",
                "authorization": "xxxxx"
              }
            }
          }
          `,
          		},
          		{
          			"config-with-passwords does not redact",
          			"config-with-passwords",
          			map[string]interface{}{
          				"config": map[string]interface{}{
          					"password": "secret",
          				},
          			},
          			`test:
          {
            "config": {
              "password": "secret"
            }

sbemiller · December 30, 2021, 4:26pm

SOLUTION
Password issue ;
I changed the password to exclude the @ and | symbol and now it works

stephenb · December 30, 2021, 4:30pm

Excellent!! You beat me to it. I had an very similar issue. I should have thought of that!!!

The @ is getting misinterpreted probably when it's entered from the command line.

You might be able to cat it in with the stdin if you really want.

cat /file/containing/setting/value | winlogbeat keystore add ES_PWD --stdin --force

system · January 27, 2022, 6:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.