Unable to create apikey in APMServer after installing 7.9

I created a new 7.9 installation on my lab environment.
I tried to create apikeys on APM Server, using documented command, but I got a couple of problems:

  1. apm-server now runs under apm-server user, and launch apikey commands under root gives error (I needed to switch user to apm-server. That's ok, I suggest to align documentation)
  2. The commands give me the following ouptut, under every condition (apm-server up/down, elasticsearch server reachable/not reachable)
-bash-4.2$ apm-server  apikey create -e -v --ingest --agent-config --name java-002
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:640    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-08-27T15:43:47.214Z        INFO    instance/beat.go:648    Beat ID: 2f71fc0a-f7a0-4a40-ac46-e556fa186810
2020-08-27T15:43:47.215Z        INFO    [config]        config/api_key.go:50    Falling back to elasticsearch output for API Key usage

elastic.output config:

output:
  elasticsearch:
    hosts: ["esnode.elk:9200"]
    protocol: https
    username: elastic
    password: ****
    ssl.certificate_authorities: ["/etc/apm-server/certs/ca.pem"]

Everything worked correctly wih 7.7.x.
Any suggestion?

Elasticsearch version: 7.9.0
APM Server version: 7.9.0

Is that the complete output? I just tested with 7.9.0 and it worked for me. I get that output, and then shortly after:

API Key created:

Name ........... java-002
Expiration ..... never
Id ............. <ID>
API Key ........ <APIKey> (won't be shown again)
Credentials .... <Credentials> (use it as "Authorization: APIKey <credentials>" header to communicate with APM Server, won't be shown again)

Are you not seeing any further output? No errors? Does the command even complete?

No further output, and the command completes immediately
I'm using Vagrant with CentOs 8.
Is it correct to execute it with apm-server user (and not root)?

Running with apm-server user is fine; what is important is that the user configured for the ES connection has the required application privileges, described in creating an API Key via APM Server.

Hi @simitt,
I reconfigured everything.
I'm able to see a go apm-agent on Kibana, but when I submit the api key command, I have:
dial tcp 127.0.0.1:9200: connect: connection refused
apm-server.yml has a different output.elasticsearch.hosts configuration (see my first post), so I don't understand why it tries to connect to localhost.

It sounds like apm-server.yml isn't being picked up. Does apm-server export config show the same output.elasticsearch configuration as in your config file?

Hi @axw,
I tested more carefully:
If apm-server.api_key is set to false I have the tcp error.
Otherwise, if it is set to true, I got success as in my first post.
Below the export config, with the correct elasticsearch host.

-bash-4.2$ apm-server export config

apm-server:
  api_key:
    enabled: false
  host: 0.0.0.0:8200
  ssl:
    certificate: /etc/apm-server/certs/apmserver.crt
    enabled: true
    key: /etc/apm-server/certs/apmserver.key
output:
  elasticsearch:
    hosts:
    - esnode.elk:9200
    password: ******
    protocol: https
    ssl:
      certificate_authorities:
      - /etc/apm-server/certs/ca.pem
    ssl_certificate:
    - /etc/apm-server/certs/apmserver.crt
    ssl_key:
    - /etc/apm-server/certs/apmserver.key
    username: elastic

I take advantage of your help and kindness.
Is there a tutorial with instructions on how to create valid apmagent key using elasticsearch rest api? (so, without using apmserver api/command line at all)
Thank you

@iorfix I'm at a loss, sorry. I just created a centos/8 box with Vagrant, and the instructions worked for me.

Config:

$ vagrant ssh -- sudo -u apm-server apm-server export config
apm-server:
  api_key:
    enabled: true
  host: localhost:8200
logging:
  files:
    rotateeverybytes: 10485760
  metrics:
    enabled: false
output:
  elasticsearch:
    hosts:
    - 192.168.121.1:9200
    password: changeme
    username: admin
path:
  config: /etc/apm-server
  data: /var/lib/apm-server
  home: /usr/share/apm-server
  logs: /var/log/apm-server
setup:
  template:
    settings:
      _source:
        enabled: true
      index:
        codec: best_compression
        mapping:
          total_fields:
            limit: 2000
        number_of_shards: 1

Connection to Elasticsearch works:

$ vagrant ssh -- sudo -u apm-server apm-server test output
elasticsearch: http://192.168.121.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.121.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.9.0

"apm-server apikey create" works:

$ vagrant ssh -- sudo -u apm-server apm-server apikey create -e -v --ingest --agent-config --name java-002
2020-09-02T02:24:46.032Z        INFO    instance/beat.go:640    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-09-02T02:24:46.032Z        INFO    instance/beat.go:648    Beat ID: a11d18fb-1631-4e91-b000-d9b0c8a9a652
2020-09-02T02:24:46.033Z        INFO    [config]        config/api_key.go:50    Falling back to elasticsearch output for API Key usage
API Key created:

Name ........... java-002
Expiration ..... never
Id ............. geehTHQBLIloR_Qkfysa
API Key ........ 31Ot-6i_QByrw_deFfjPpA (won't be shown again)
Credentials .... Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ== (use it as "Authorization: APIKey <credentials>" header to communicate with APM Server, won't be shown again)

Credentials work:

$ curl -H "Authorization: ApiKey Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ==" http://192.168.121.1:9200/_security/_authenticate?pretty
{
  "username" : "admin",
  "roles" : [ ],
  "full_name" : null,
  "email" : null,
  "metadata" : { },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "_es_api_key",
    "type" : "_es_api_key"
  },
  "lookup_realm" : {
    "name" : "_es_api_key",
    "type" : "_es_api_key"
  }
}

Hi @axw,
I resolved the issue.
I had an incorrect proxy configuration with no_proxy for "192.168.*", but not for "*.elk.svi".
The error was quite subtle, since o.s. ping worked correctly, and apmserver too!
Thank you for helping me in the issue, since I discovered the problem using the test output command (see incorrect output below).
Anyway, is there a way to create the couple Api Key:Credentials directly using Elasticseatch Rest Api, and not apmserver?

[vagrant@apmserver ~]$ sudo -u apm-server apm-server test output -v -e -d "*"
2020-09-02T08:19:07.470Z        INFO    instance/beat.go:640    Home path: [/usr        /share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-serve        r] Logs path: [/var/log/apm-server]
2020-09-02T08:19:07.470Z        DEBUG   [beat]  instance/beat.go:692    Beat met        adata path: /var/lib/apm-server/meta.json
2020-09-02T08:19:07.470Z        INFO    instance/beat.go:648    Beat ID: 344fff2        4-8338-4285-9377-ca4ecefbaeec
2020-09-02T08:19:07.470Z        INFO    [index-management]      idxmgmt/std.go:1        84      Set output.elasticsearch.index to 'apm-server-7.9.0' as ILM is enabled.
2020-09-02T08:19:07.471Z        DEBUG   [tls]   tlscommon/tls.go:155    tls%!(EX        TRA string=successfully loaded CA certificate: %v, string=/etc/apm-server/certs/        ca.pem)
2020-09-02T08:19:07.471Z        INFO    eslegclient/connection.go:99    elastics        earch url: https://esnode.elk.svi:9200
elasticsearch: https://esnode.elk.svi:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.9.90
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
2020-09-02T08:19:18.537Z        DEBUG   [esclientleg]   eslegclient/connection.g        o:290   ES Ping(url=https://esnode.elk.svi:9200)
2020-09-02T08:19:18.545Z        DEBUG   [esclientleg]   eslegclient/connection.g        o:294   Ping request failed with: Get https://esnode.elk.svi:9200: Success
  talk to server... ERROR Get https://esnode.elk.svi:9200: Success

[vagrant@apmserver ~]$ ping esnode1.elk.svi
PING esnode1.elk.svi (192.168.9.90) 56(84) bytes of data.
64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=1 ttl=64 time=0.898 ms
64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=2 ttl=64 time=0.787 ms
1 Like

Hooray! That was a subtle issue indeed.

You can use the REST API directly:

POST /_security/api_key
{
  "name": "my-api-key",
  "expiration": "1d", 
  "role_descriptors": { 
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}

This produces something like

{
  "id" : "l6YgTnQBiCXtW5azhjNu",
  "name" : "my-api-key",
  "expiration" : 1599124988514,
  "api_key" : "iyPzlznbQmmgY3mSP3M3FA"
}

To form the "credentials" string, you need to base-64 encode <id>:<api_key>, like:

echo -n l6YgTnQBiCXtW5azhjNu:iyPzlznbQmmgY3mSP3M3FA | base64
bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ==

You can verify this with the apm-server CLI:

$ ./apm-server --strict.perms=false apikey verify --credentials=bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ==
Authorized for privilege "config_agent:read"...: Yes
Authorized for privilege "event:write"...: Yes
Authorized for privilege "sourcemap:write"...: Yes

Thank you,
it works.
I suggest to add this instructions on apmserver documentation. I haven't find it elsewhere

@iorfix sounds like a good idea. I've created an issue to track this: https://github.com/elastic/apm-server/issues/4135

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.