Secure Communications from Beats to Elasticsearch

Elastic Stack Elasticsearch
1

Using Filebeat's article on ... found here as reference, we see that it says that "Filebeat can use any of the following authentication methods". 3 authentication methods were then listed.

My question is: are we only able to use PKI certificates as the sole authentication method?

I was examining an existing setup done by another colleague (who left), who had both PKI certificate AND token-based API authentication in the Beats yml file. When I commented out one or another and restarted the Beats, it no longer works.

I created a brand new setup altogether and tried using only PKI certificates as authentication and got the error "missing authentication credentials for REST request".

2

Sharing the config and errors you saw would be helpful.

3

Sorry for the lack of details, here goes:

This is a snippet when I ran .\winlogbeat -e:

2020-11-16T09:15:15.195+0800    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://[IP address]:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

This is the relevant part in winlogbeat.yml:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["[IP address]:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"
  
  ssl.certficate_authorities: ['C:\Program Files\Elastic\winlogbeat\ca.crt']
  ssl.certificate: C:\Program Files\Elastic\winlogbeat\master-node.crt
  ssl.key: C:\Program Files\Elastic\winlogbeat\master-node.key
  ssl.verification_mode: none

When I added a valid username and password, or valid api_key, it works fine.

4

This core of the problem here is terminology - Elasticsearch use specific terms with specific meanings here, but it's hard to capture that nuance in a summary document like the one you refer to.
Your use of "PKI" in the description of what you inherited, is not quite accurate in Elasticsearch terms.

In Elasticsearch:

  • SSL client authentication refers to the use of client-side X.509 certificates as part of SSL/TLS handshaking. If you set the xpack.security.http.ssl.client_authentication setting to required then all connections to the cluster must provide a valid, trusted client certificate. This is what is commonly referred to as "mutual TLS" (sometimes mTLS). By itself, that is a network based control only. It does not integrate with Elasticsearch's internal RBAC security model.
  • PKI authentication refers to using client-side X.509 certificates as a mechanism to authenticate Elasticsearch users. It is an alternative to providing a username & password, or an API key for authentication.

Within that Elasticsearch terminology, the use of client certificates for TLS is not, by itself, "PKI", it is "mutual TLS" or "SSL client authentication". You can use those certificates to implement PKI authentication, but they are distinct terms & concepts.

I was examining an existing setup done by another colleague (who left), who had both PKI certificate AND token-based API authentication in the Beats yml file

Strictly speaking, from a Beat yml file, you can only tell that it provided client certificates, you cannot tell whether they are used for "PKI" without looking at the Elasticsearch configuration. The elasticsearch.yml will determine whether SSL client certificates can be used for PKI based integration into the RBAC model, or are merely used as a network control.

Assuming that you used the same client certificate & key that works for the existing beats setup, this implies that the Elasticsearch server is set up for SSL client authentication but not PKI.
However there are other explanations, so if you have access to the elasticsearch.yml config, that will help narrow it down.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.