Unable to create apikey in APMServer after installing 7.9

iorfix · August 27, 2020, 4:03pm

I created a new 7.9 installation on my lab environment.
I tried to create apikeys on APM Server, using documented command, but I got a couple of problems:

apm-server now runs under apm-server user, and launch apikey commands under root gives error (I needed to switch user to apm-server. That's ok, I suggest to align documentation)
The commands give me the following ouptut, under every condition (apm-server up/down, elasticsearch server reachable/not reachable)

-bash-4.2$ apm-server  apikey create -e -v --ingest --agent-config --name java-002
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:640    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:648    Beat ID: 2f71fc0a-f7a0-4a40-ac46-e556fa186810
2020-08-27T15:43:47.215Z        INFO    [config]        config/api_key.go:50    Falling back to elasticsearch output for API Key usage

elastic.output config:

output:
  elasticsearch:
    hosts: ["esnode.elk:9200"]
    protocol: https
    username: elastic
    password: ****
    ssl.certificate_authorities: ["/etc/apm-server/certs/ca.pem"]

Everything worked correctly wih 7.7.x.
Any suggestion?

Elasticsearch version: 7.9.0
APM Server version: 7.9.0

axw · August 31, 2020, 2:37am

-bash-4.2$ apm-server  apikey create -e -v --ingest --agent-config --name java-002
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:640    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:648    Beat ID: 2f71fc0a-f7a0-4a40-ac46-e556fa186810
2020-08-27T15:43:47.215Z        INFO    [config]        config/api_key.go:50    Falling back to elasticsearch output for API Key usage

Is that the complete output? I just tested with 7.9.0 and it worked for me. I get that output, and then shortly after:

API Key created:

Name ........... java-002
Expiration ..... never
Id ............. <ID>
API Key ........ <APIKey> (won't be shown again)
Credentials .... <Credentials> (use it as "Authorization: APIKey <credentials>" header to communicate with APM Server, won't be shown again)

Are you not seeing any further output? No errors? Does the command even complete?

iorfix · August 31, 2020, 11:05am

No further output, and the command completes immediately
I'm using Vagrant with CentOs 8.
Is it correct to execute it with apm-server user (and not root)?

simitt · August 31, 2020, 4:06pm

Running with apm-server user is fine; what is important is that the user configured for the ES connection has the required application privileges, described in creating an API Key via APM Server.

iorfix · September 1, 2020, 7:43am

Hi @simitt,
I reconfigured everything.
I'm able to see a go apm-agent on Kibana, but when I submit the api key command, I have:
dial tcp 127.0.0.1:9200: connect: connection refused
apm-server.yml has a different output.elasticsearch.hosts configuration (see my first post), so I don't understand why it tries to connect to localhost.

axw · September 1, 2020, 8:00am

It sounds like apm-server.yml isn't being picked up. Does apm-server export config show the same output.elasticsearch configuration as in your config file?

iorfix · September 1, 2020, 8:12am

Hi @axw,
I tested more carefully:
If apm-server.api_key is set to false I have the tcp error.
Otherwise, if it is set to true, I got success as in my first post.
Below the export config, with the correct elasticsearch host.

-bash-4.2$ apm-server export config

apm-server:
  api_key:
    enabled: false
  host: 0.0.0.0:8200
  ssl:
    certificate: /etc/apm-server/certs/apmserver.crt
    enabled: true
    key: /etc/apm-server/certs/apmserver.key
output:
  elasticsearch:
    hosts:
    - esnode.elk:9200
    password: ******
    protocol: https
    ssl:
      certificate_authorities:
      - /etc/apm-server/certs/ca.pem
    ssl_certificate:
    - /etc/apm-server/certs/apmserver.crt
    ssl_key:
    - /etc/apm-server/certs/apmserver.key
    username: elastic

iorfix · September 1, 2020, 9:25am

I take advantage of your help and kindness.
Is there a tutorial with instructions on how to create valid apmagent key using elasticsearch rest api? (so, without using apmserver api/command line at all)
Thank you

axw · September 2, 2020, 2:28am

@iorfix I'm at a loss, sorry. I just created a centos/8 box with Vagrant, and the instructions worked for me.

Config:

$ vagrant ssh -- sudo -u apm-server apm-server export config
apm-server:
  api_key:
    enabled: true
  host: localhost:8200
logging:
  files:
    rotateeverybytes: 10485760
  metrics:
    enabled: false
output:
  elasticsearch:
    hosts:
    - 192.168.121.1:9200
    password: changeme
    username: admin
path:
  config: /etc/apm-server
  data: /var/lib/apm-server
  home: /usr/share/apm-server
  logs: /var/log/apm-server
setup:
  template:
    settings:
      _source:
        enabled: true
      index:
        codec: best_compression
        mapping:
          total_fields:
            limit: 2000
        number_of_shards: 1

Connection to Elasticsearch works:

$ vagrant ssh -- sudo -u apm-server apm-server test output
elasticsearch: http://192.168.121.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.121.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.9.0

"apm-server apikey create" works:

$ vagrant ssh -- sudo -u apm-server apm-server apikey create -e -v --ingest --agent-config --name java-002
2020-09-02T02:24:46.032Z        INFO    instance/beat.go:640    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-09-02T02:24:46.032Z        INFO    instance/beat.go:648    Beat ID: a11d18fb-1631-4e91-b000-d9b0c8a9a652
2020-09-02T02:24:46.033Z        INFO    [config]        config/api_key.go:50    Falling back to elasticsearch output for API Key usage
API Key created:

Name ........... java-002
Expiration ..... never
Id ............. geehTHQBLIloR_Qkfysa
API Key ........ 31Ot-6i_QByrw_deFfjPpA (won't be shown again)
Credentials .... Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ== (use it as "Authorization: APIKey <credentials>" header to communicate with APM Server, won't be shown again)

Credentials work:

$ curl -H "Authorization: ApiKey Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ==" http://192.168.121.1:9200/_security/_authenticate?pretty
{
  "username" : "admin",
  "roles" : [ ],
  "full_name" : null,
  "email" : null,
  "metadata" : { },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "_es_api_key",
    "type" : "_es_api_key"
  },
  "lookup_realm" : {
    "name" : "_es_api_key",
    "type" : "_es_api_key"
  }
}

iorfix · September 2, 2020, 8:13am

Hi @axw,
I resolved the issue.
I had an incorrect proxy configuration with no_proxy for "192.168.*", but not for "*.elk.svi".
The error was quite subtle, since o.s. ping worked correctly, and apmserver too!
Thank you for helping me in the issue, since I discovered the problem using the test output command (see incorrect output below).
Anyway, is there a way to create the couple Api Key:Credentials directly using Elasticseatch Rest Api, and not apmserver?

[vagrant@apmserver ~]$ sudo -u apm-server apm-server test output -v -e -d "*"
2020-09-02T08:19:07.470Z        INFO    instance/beat.go:640    Home path: [/usr        /share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-serve        r] Logs path: [/var/log/apm-server]
2020-09-02T08:19:07.470Z        DEBUG   [beat]  instance/beat.go:692    Beat met        adata path: /var/lib/apm-server/meta.json
2020-09-02T08:19:07.470Z        INFO    instance/beat.go:648    Beat ID: 344fff2        4-8338-4285-9377-ca4ecefbaeec
2020-09-02T08:19:07.470Z        INFO    [index-management]      idxmgmt/std.go:1        84      Set output.elasticsearch.index to 'apm-server-7.9.0' as ILM is enabled.
2020-09-02T08:19:07.471Z        DEBUG   [tls]   tlscommon/tls.go:155    tls%!(EX        TRA string=successfully loaded CA certificate: %v, string=/etc/apm-server/certs/        ca.pem)
2020-09-02T08:19:07.471Z        INFO    eslegclient/connection.go:99    elastics        earch url: https://esnode.elk.svi:9200
elasticsearch: https://esnode.elk.svi:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.9.90
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
2020-09-02T08:19:18.537Z        DEBUG   [esclientleg]   eslegclient/connection.g        o:290   ES Ping(url=https://esnode.elk.svi:9200)
2020-09-02T08:19:18.545Z        DEBUG   [esclientleg]   eslegclient/connection.g        o:294   Ping request failed with: Get https://esnode.elk.svi:9200: Success
  talk to server... ERROR Get https://esnode.elk.svi:9200: Success

[vagrant@apmserver ~]$ ping esnode1.elk.svi
PING esnode1.elk.svi (192.168.9.90) 56(84) bytes of data.
64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=1 ttl=64 time=0.898 ms
64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=2 ttl=64 time=0.787 ms

axw · September 2, 2020, 9:28am

Hooray! That was a subtle issue indeed.

You can use the REST API directly:

POST /_security/api_key
{
  "name": "my-api-key",
  "expiration": "1d", 
  "role_descriptors": { 
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}

This produces something like

{
  "id" : "l6YgTnQBiCXtW5azhjNu",
  "name" : "my-api-key",
  "expiration" : 1599124988514,
  "api_key" : "iyPzlznbQmmgY3mSP3M3FA"
}

To form the "credentials" string, you need to base-64 encode <id>:<api_key>, like:

echo -n l6YgTnQBiCXtW5azhjNu:iyPzlznbQmmgY3mSP3M3FA | base64
bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ==

You can verify this with the apm-server CLI:

$ ./apm-server --strict.perms=false apikey verify --credentials=bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ==
Authorized for privilege "config_agent:read"...: Yes
Authorized for privilege "event:write"...: Yes
Authorized for privilege "sourcemap:write"...: Yes

iorfix · September 2, 2020, 9:38am

Thank you,
it works.
I suggest to add this instructions on apmserver documentation. I haven't find it elsewhere

axw · September 2, 2020, 9:48am

@iorfix sounds like a good idea. I've created an issue to track this: https://github.com/elastic/apm-server/issues/4135

system · September 23, 2020, 5:48am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.