PKI authentication issue for xpack monitoring exporter

Elastic Stack Elasticsearch
1

Hello. I want to set monitoring exporter on my prod ES cluster to send monitoring data to my monitoring cluster. I enable PKI authentication on my monitoring cluster and I am able to authenticate with my client certificate to auth api via curl:

curl -k --cert-type pem --cert ./ctest.pem https://emon1.my.domain:9210/_xpack/security/_authenticate?pretty
{
  "username" : "emon1.my.domain",
  "roles" : [
    "remote_monitoring_agent"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=emon1.my.domain, OU=someou, O=someo, L=somelocality, ST=somestate, C=somecountry"
  },
  "enabled" : true
}

But PKI auth for xpack exporter does not work. Settings:

xpack:
  monitoring:
    exporters:
      my_emon:
        type: http
        host: [ "${ELASTICSEARCH_CONFIGVAR_ESMON_SERVER_URL_1}" ]
        ssl:
          certificate_authorities: [ "${ELASTICSEARCH_CONFIG_PATH}/certs/ca.crt" ]
          supported_protocols: [ "TLSv1.2" ]
          verification_method: "full"
          certificate: "${ELASTICSEARCH_CONFIG_PATH}/certs/client.crt"
          key: "${ELASTICSEARCH_CONFIG_PATH}/certs/client.key"

Auth fails, here is prod ES cluster log:

monitoring cluster with status code [401]
org.elasticsearch.client.ResponseException: GET https://emon1.my.domain:9210/_template/.monitoring-es?filter_path=*.version: HTTP/1.1 401 Unauthorized
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_template/.monitoring-es?filter_path=*.version]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_template/.monitoring-es?filter_path=*.version]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

What is wrong with my configuration? Or is that a bug in X-Pack (I am using version 5.5)?

2

It looks like a X-Pack bug. I found a way to workaround it. First of all I set full cluster privileges for x-pack exporter user. I did it via role mapping API. But in fact it still did not work until I touched the authenticated API via curl on all cluster nodes:

curl -k --cert-type pem --cert /root/clientcert -X GET https://emon1.my.domain:9210/_xpack/security/_authenticate
curl -k --cert-type pem --cert /root/clientcert -X GET https://emon2.my.domain:9210/_xpack/security/_authenticate
curl -k --cert-type pem --cert /root/clientcert -X GET https://emon2.my.domain:9210/_xpack/security/_authenticate

After that no error message in elasticsearch log anymore. I can confirm it using curl itself. If I try this one first:

curl -k --cert-type pem --cert /root/clientcert -X GET https://emon2.my.domain:9210_template/.monitoring-es?filter_path=*.version

I get "missing authentication token for REST request" but

curl -k --cert-type pem --cert /root/clientcert -X GET https://emon1.my.domain:9210/_xpack/security/_authenticate
curl -k --cert-type pem --cert /root/clientcert -X GET https://emon2.my.domain:9210_template/.monitoring-es?filter_path=*.version

worked. You can see first I had to touch the authenticate API.

This looks like a bug. Is it?

3

That seems unlikely. The authenticate API doesn't do anything special. If this is indeed what happened, then it is more likely that it was some sort of caching issue, and the cache expired at just the right time.

This is a broken URL. If that actually what you sent, then I'm surprised it worked at all. If it's not what you sent, can you please post your commands and logs exactly as they happened. It's impossible for me to try and reproduce the problem if I don't have the correct information.

Thanks.

4

Maybe it is cache related, but the cache has been refreshed just adter calling this particular URL (it worker for each server of my cluster).

This has been actually taken from the error message (see "Auth fails, here is prod ES cluster log:" of the first post for details)

5

Somewhere along the way it has been modified from the URL in the error message.

The error has:

But you said you were using:

Note the missing / between the port number and the path.
You've also changed the hostname, which might be fine, but is potentially a distraction from the issue we're trying to solve. It will be simpler to resolve the problem if we keep everything on a single node.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.