APM Server -> AWS Elasticsearch Service: "cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials"

Elastic Observability APM
1

Kibana version: 7.1.1

Elasticsearch version: 7.1.1

APM Server version: 7.1.1

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
DEB:

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-7.1.1-amd64.deb
sudo dpkg -i apm-server-7.1.1-amd64.deb

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup?
It's AWS's elasticsearch service.
apm-server is running on an ubuntu 18.04 instance. They're in the same VPC.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I may have some misconceptions or issues with my setup, so please correct me if I'm wrong on any of this.

The log output, trying to connect to my elasticsearch instance displays:

Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials

It appears that AWS runs on port 443 instead of 9200:

curl -XGET 'https://<snip>.us-east-1.es.amazonaws.com:443'

Yields:

      "name" : "<snip>",
      "cluster_name" : "<snip>",
      "cluster_uuid" : "<snip>",
      "version" : {
        "number" : "7.1.1",
        "build_flavor" : "oss",
        "build_type" : "tar",
        "build_hash" : "7a013de",
        "build_date" : "2019-09-05T07:25:23.525600Z",
        "build_snapshot" : false,
        "lucene_version" : "8.0.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }

Does anyone know what I may be doing wrong here?

2

APM is not compatible with the aws service unfortunately.

Try https://www.elastic.co/products/elasticsearch/service instead :slight_smile:

3

This thread seems to say that it's possible just by putting in the hostname of the aws instance and proper port.

Has that compatibility been lost along the way?

4

Ahh right, I missed that nuance sorry!

5

Well, I guess what I'm wondering is what's the process for getting that to actually happen? I've set things up as it was stated there:
To configure AWS Server to send data to a remote Elasticseach instance (as is the case with AWS ES), you need to specify the hostname/port of the AWS ES cluster using the output.elasticsearch.hosts config option.

But...that led me to creating this thread in the first place.

6

@hickeroar You'll need to use the oss distribution of APM Server when using AWS' hosted service, https://artifacts.elastic.co/downloads/apm-server/apm-server-oss-7.1.1-amd64.deb in your case. Any reason you have to use 7.1.1? We'd recommend the latest release if possible, it's compatible with ES 7.1.1 in case you are pinned to that version.

We've recently updated the error message to make it more clear:

Connection marked as failed because the onConnect callback failed: Apm-Server requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Apm-Server.
3 Likes
7

OK thank you. Got it all sorted now.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.