Error connecting to Elasticsearch after upgrading to Heartbeat v6.7.0

Elastic Stack Beats
1

I recently attempted to update one of my servers running Heartbeat v6.6.2 to v6.7.0 and started receiving the following error message:

INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(https://elasticsearch.example.com:443))
INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.4.2
pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://elasticsearch.example.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials

As a further check, I ran the heartbeat test command to test the connection to elasticsearch and received the following:

$ heartbeat test output -c heartbeat.yml
elasticsearch: https://elasticsearch.example.com:443...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 3.86.58.90
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Note: I replaced my elasticsearch dns with elasticsearch.example.com above.

I have NGINX running in front of my Elasticsearch cluster which handles offloading basic authentication. I am able to replicate the problem with this minimal heartbeat.yml

heartbeat.monitors:
  - type: icmp
    hosts: ["localhost"]
    schedule: "@every 10s"
output.elasticsearch:
  hosts: ["https://elasticsearch.example.com:443"]
  username: "my_basic_auth_username"
  password: "my_basic_auth_password"

I believe that the problem is a result of this update: https://github.com/elastic/beats/pull/11296

Is there a reason Heartbeat now checks for an x-pack license? Is there a way to disable this functionality?

2

Hi,

I will assume you are running the default elasticsearch distribution and not the OSS one because if you were using the OSS distribution across the board you wouldn't have that issue.
(Not saying that's what you should do, since you will loose all the free basic features included in the default distro.)

You'll need to allow the /_xpack endpoint on your NGINX running in front of your ES cluster.

Beats >6.7 from the default distribution now check the license on ES.

Your NGINX is blocking the call to elasticsearch.example.com/_xpack and answering with an http authorization failure.
I'm guessing you only allow a subset of whitelisted endpoints in your NGINX config where you implemented authorization. bulk, index, cluster etc.
Do the same with /_xpack (could also be /_xpack/license but the code I checked in Elastic repo seem to point to /_xpack)

If not, or if you still have issues I recommend you add details like which distribution(default or OSS) of beats and elasticsearch you're using and the version of your ES.
You can also try elasticsearch.example.com/_xpack from your browser just like you should be able to do elasticsearch.example.com/_cluster/health.
In both cases, based on your description of your NGINX setup you should get prompted for auth and then see a JSON response, for both.
Currently your error indicate that /_xpack only returns unauthorized, which means beats throw this error you have.

Martin

1 Like
3

From a debug log:

DEBUG   [elasticsearch] elasticsearch/client.go:757     GET https://elasticsearch.example.com:443/_xpack?human=false  <nil>

So it means it is calling /_xpack?human=false .
Wanted to add this tidbit I found while in debug.

Martin

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.