Why metricbeat is not picking up keystore entries?

[elasticheart]

Hi,

I am using elastic stack 7.9. I have a 3 node cluster and https is I have created certificates using elasticsearch-certutil command, which are generic, and I am using it across all nodes. I have the below files;

1. elastic-stack-ca.p12 (common for all nodes)
2. elastic-certificates.p12 (common for all nodes)
3. http.p12 (for https)
4. elasticsearch-ca.pem (for kibana)

Now, I am trying to integrate metricbeat. I have provided the below entries in metricbeat keystore;

setup.kibana.username
setup.kibana.password
output.elasticsearch.username
output.elasticsearch.password
monitoring.elasticsearch.username
monitoring.elasticsearch.password

When I try to setup metricbeat, using the command metricbeat setup -e, I am getting the below error;

2020-11-19T13:33:11.161Z        ERROR   [esclientleg]   eslegclient/connection.go:261   error connecting to Elasticsearch at https://192.168.1.1:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
2020-11-19T13:33:11.175Z        ERROR   [esclientleg]   eslegclient/connection.go:261   error connecting to Elasticsearch at https://192.168.1.2:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
2020-11-19T13:33:11.187Z        ERROR   [esclientleg]   eslegclient/connection.go:261   error connecting to Elasticsearch at https://192.168.1.3:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
2020-11-19T13:33:11.187Z        ERROR   instance/beat.go:951    Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://192.168.1.1:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401} error connecting to Elasticsearch at https://192.168.1.2:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401} error connecting to Elasticsearch at https://192.168.1.3:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://192.168.1.1:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401} error connecting to Elasticsearch at https://192.168.1.2:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401} error connecting to Elasticsearch at https://192.168.1.3:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}]

But, if I provide;

output.elasticsearch:
  username: "elastic"
  password: "password"

This works like charm. Why is this happening? Is there anywhere should I mention the p12 files?

Thanks.

[Wolfram_Haussig]

Hello,

Beats do not automatically check for settings in keystore as ElasticSearch does. To solve your problem you need to add the settings to your yml file and reference the keys from your keystore:

output.elasticsearch:
  username: "${output.elasticsearch.username}"
  password: "${output.elasticsearch.password}"

I prefer this way over the ElasticSearch way as it is more readable (you do not have to assume that settings missing here are stored in the keystore) and you can rename the keys in the keystore to better describer what they contain(e.g. output.prod-cluster-1.username instead ofoutput.elasticsearch.username).

Best regards
Wolfram

[elasticheart]

Hi @Wolfram_Haussig , thank you for your time.

This is really helpful information and a handy tip as well.
It works!

Best regards,
Alfred