# Unable to create apikey in APMServer after installing 7.9 **URL:** \ **Category:** APM\ **Tags:** server\ **Created:** [August 27, 2020, 4:03pm UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659 "2020-08-27T16:03:13Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [August 27, 2020, 4:03pm UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/1 "2020-08-27T16:03:13Z")
I created a new 7.9 installation on my lab environment. I tried to create apikeys on APM Server, using documented command, but I got a couple of problems: 1. apm-server now runs under apm-server user, and launch apikey commands under root gives error (I needed to switch user to apm-server. That's ok, I suggest to align documentation) 2. The commands give me the following ouptut, under every condition (apm-server up/down, elasticsearch server reachable/not reachable) ```auto -bash-4.2$ apm-server apikey create -e -v --ingest --agent-config --name java-002 2020-08-27T15:43:47.214Z INFO instance/beat.go:640 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server] 2020-08-27T15:43:47.214Z INFO instance/beat.go:648 Beat ID: 2f71fc0a-f7a0-4a40-ac46-e556fa186810 2020-08-27T15:43:47.215Z INFO [config] config/api_key.go:50 Falling back to elasticsearch output for API Key usage ``` elastic.output config: ```auto output: elasticsearch: hosts: ["esnode.elk:9200"] protocol: https username: elastic password: **** ssl.certificate_authorities: ["/etc/apm-server/certs/ca.pem"] ``` Everything worked correctly wih 7.7.x. Any suggestion? **Elasticsearch version** : 7.9.0 **APM Server version** : 7.9.0 ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [August 31, 2020, 2:37am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/2 "2020-08-31T02:37:41Z")
> [@](#): > > ```auto > -bash-4.2$ apm-server apikey create -e -v --ingest --agent-config --name java-002 > 2020-08-27T15:43:47.214Z INFO instance/beat.go:640 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server] > 2020-08-27T15:43:47.214Z INFO instance/beat.go:648 Beat ID: 2f71fc0a-f7a0-4a40-ac46-e556fa186810 > 2020-08-27T15:43:47.215Z INFO [config] config/api_key.go:50 Falling back to elasticsearch output for API Key usage > > ``` Is that the complete output? I just tested with 7.9.0 and it worked for me. I get that output, and then shortly after: ```auto API Key created: Name ........... java-002 Expiration ..... never Id ............. API Key ........ (won't be shown again) Credentials .... (use it as "Authorization: APIKey " header to communicate with APM Server, won't be shown again) ``` Are you not seeing any further output? No errors? Does the command even complete? ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [August 31, 2020, 11:05am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/3 "2020-08-31T11:05:26Z")
No further output, and the command completes immediately I'm using Vagrant with CentOs 8. Is it correct to execute it with apm-server user (and not root)? ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [August 31, 2020, 4:06pm UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/4 "2020-08-31T16:06:10Z")
Running with apm-server user is fine; what is important is that the user configured for the ES connection has the required application privileges, described in [creating an API Key via APM Server](https://www.elastic.co/guide/en/apm/server/7.9/api-key.html#create-api-key). ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [September 1, 2020, 7:43am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/5 "2020-09-01T07:43:32Z")
Hi @simitt, I reconfigured everything. I'm able to see a go apm-agent on Kibana, but when I submit the api key command, I have: `dial tcp 127.0.0.1:9200: connect: connection refused` apm-server.yml has a different output.elasticsearch.hosts configuration (see my first post), so I don't understand why it tries to connect to localhost. ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 1, 2020, 8:00am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/6 "2020-09-01T08:00:41Z")
It sounds like `apm-server.yml` isn't being picked up. Does `apm-server export config` show the same `output.elasticsearch` configuration as in your config file? ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [September 1, 2020, 8:12am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/7 "2020-09-01T08:12:28Z")
Hi @axw, I tested more carefully: If _apm-server.api\_key_ is set to _false_ I have the tcp error. Otherwise, if it is set to _true_, I got _success_ as in my first post. Below the export config, with the correct elasticsearch host. `-bash-4.2$ apm-server export config` ```auto apm-server: api_key: enabled: false host: 0.0.0.0:8200 ssl: certificate: /etc/apm-server/certs/apmserver.crt enabled: true key: /etc/apm-server/certs/apmserver.key output: elasticsearch: hosts: - esnode.elk:9200 password: ****** protocol: https ssl: certificate_authorities: - /etc/apm-server/certs/ca.pem ssl_certificate: - /etc/apm-server/certs/apmserver.crt ssl_key: - /etc/apm-server/certs/apmserver.key username: elastic ``` ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [September 1, 2020, 9:25am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/8 "2020-09-01T09:25:37Z")
I take advantage of your help and kindness. Is there a tutorial with instructions on how to create valid apmagent key using elasticsearch rest api? (so, without using apmserver api/command line at all) Thank you ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 2, 2020, 2:28am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/9 "2020-09-02T02:28:46Z")
@iorfix I'm at a loss, sorry. I just created a centos/8 box with Vagrant, and the instructions worked for me. Config: ```auto $ vagrant ssh -- sudo -u apm-server apm-server export config apm-server: api_key: enabled: true host: localhost:8200 logging: files: rotateeverybytes: 10485760 metrics: enabled: false output: elasticsearch: hosts: - 192.168.121.1:9200 password: changeme username: admin path: config: /etc/apm-server data: /var/lib/apm-server home: /usr/share/apm-server logs: /var/log/apm-server setup: template: settings: _source: enabled: true index: codec: best_compression mapping: total_fields: limit: 2000 number_of_shards: 1 ``` Connection to Elasticsearch works: ```auto $ vagrant ssh -- sudo -u apm-server apm-server test output elasticsearch: http://192.168.121.1:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 192.168.121.1 dial up... OK TLS... WARN secure connection disabled talk to server... OK version: 7.9.0 ``` "apm-server apikey create" works: ```auto $ vagrant ssh -- sudo -u apm-server apm-server apikey create -e -v --ingest --agent-config --name java-002 2020-09-02T02:24:46.032Z INFO instance/beat.go:640 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server] 2020-09-02T02:24:46.032Z INFO instance/beat.go:648 Beat ID: a11d18fb-1631-4e91-b000-d9b0c8a9a652 2020-09-02T02:24:46.033Z INFO [config] config/api_key.go:50 Falling back to elasticsearch output for API Key usage API Key created: Name ........... java-002 Expiration ..... never Id ............. geehTHQBLIloR_Qkfysa API Key ........ 31Ot-6i_QByrw_deFfjPpA (won't be shown again) Credentials .... Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ== (use it as "Authorization: APIKey " header to communicate with APM Server, won't be shown again) ``` Credentials work: ```auto $ curl -H "Authorization: ApiKey Z2VlaFRIUUJMSWxvUl9Ra2Z5c2E6MzFPdC02aV9RQnlyd19kZUZmalBwQQ==" http://192.168.121.1:9200/_security/_authenticate?pretty { "username" : "admin", "roles" : [], "full_name" : null, "email" : null, "metadata" : { }, "enabled" : true, "authentication_realm" : { "name" : "_es_api_key", "type" : "_es_api_key" }, "lookup_realm" : { "name" : "_es_api_key", "type" : "_es_api_key" } } ``` ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [September 2, 2020, 8:13am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/10 "2020-09-02T08:13:10Z")
Hi @axw, I resolved the issue. I had an incorrect proxy configuration with no\_proxy for "192.168.\*", but not for "\*.elk.svi". The error was quite subtle, since o.s. ping worked correctly, and apmserver too! Thank you for helping me in the issue, since I discovered the problem using the _test output_ command (see incorrect output below). Anyway, is there a way to create the couple _Api Key:Credentials_ directly using Elasticseatch Rest Api, and not apmserver? ```auto [vagrant@apmserver ~]$ sudo -u apm-server apm-server test output -v -e -d "*" 2020-09-02T08:19:07.470Z INFO instance/beat.go:640 Home path: [/usr /share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-serve r] Logs path: [/var/log/apm-server] 2020-09-02T08:19:07.470Z DEBUG [beat] instance/beat.go:692 Beat met adata path: /var/lib/apm-server/meta.json 2020-09-02T08:19:07.470Z INFO instance/beat.go:648 Beat ID: 344fff2 4-8338-4285-9377-ca4ecefbaeec 2020-09-02T08:19:07.470Z INFO [index-management] idxmgmt/std.go:1 84 Set output.elasticsearch.index to 'apm-server-7.9.0' as ILM is enabled. 2020-09-02T08:19:07.471Z DEBUG [tls] tlscommon/tls.go:155 tls%!(EX TRA string=successfully loaded CA certificate: %v, string=/etc/apm-server/certs/ ca.pem) 2020-09-02T08:19:07.471Z INFO eslegclient/connection.go:99 elastics earch url: https://esnode.elk.svi:9200 elasticsearch: https://esnode.elk.svi:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 192.168.9.90 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK 2020-09-02T08:19:18.537Z DEBUG [esclientleg] eslegclient/connection.g o:290 ES Ping(url=https://esnode.elk.svi:9200) 2020-09-02T08:19:18.545Z DEBUG [esclientleg] eslegclient/connection.g o:294 Ping request failed with: Get https://esnode.elk.svi:9200: Success talk to server... ERROR Get https://esnode.elk.svi:9200: Success [vagrant@apmserver ~]$ ping esnode1.elk.svi PING esnode1.elk.svi (192.168.9.90) 56(84) bytes of data. 64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=1 ttl=64 time=0.898 ms 64 bytes from esnode1.elk.svi (192.168.9.90): icmp_seq=2 ttl=64 time=0.787 ms ``` ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 2, 2020, 9:28am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/11 "2020-09-02T09:28:05Z")
> [@iorfix](#): > > I resolved the issue. Hooray! That was a subtle issue indeed. You can use the [REST API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html) directly: ```auto POST /_security/api_key { "name": "my-api-key", "expiration": "1d", "role_descriptors": { "apm": { "applications": [ { "application": "apm", "privileges": ["sourcemap:write", "event:write", "config_agent:read"], "resources": ["*"] } ] } } } ``` This produces something like ```auto { "id" : "l6YgTnQBiCXtW5azhjNu", "name" : "my-api-key", "expiration" : 1599124988514, "api_key" : "iyPzlznbQmmgY3mSP3M3FA" } ``` To form the "credentials" string, you need to base-64 encode `:`, like: ```auto echo -n l6YgTnQBiCXtW5azhjNu:iyPzlznbQmmgY3mSP3M3FA | base64 bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ== ``` You can verify this with the apm-server CLI: ```auto $ ./apm-server --strict.perms=false apikey verify --credentials=bDZZZ1RuUUJpQ1h0VzVhemhqTnU6aXlQemx6bmJRbW1nWTNtU1AzTTNGQQ== Authorized for privilege "config_agent:read"...: Yes Authorized for privilege "event:write"...: Yes Authorized for privilege "sourcemap:write"...: Yes ``` ---
**Author:** ![iorfix](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@iorfix](https://discuss.elastic.co/u/iorfix)\ **Post date:** [September 2, 2020, 9:38am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/12 "2020-09-02T09:38:05Z")
Thank you, it works. I suggest to add this instructions on apmserver documentation. I haven't find it elsewhere ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 2, 2020, 9:48am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/13 "2020-09-02T09:48:56Z")
@iorfix sounds like a good idea. I've created an issue to track this: [https://github.com/elastic/apm-server/issues/4135](https://github.com/elastic/apm-server/issues/4135) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 23, 2020, 5:48am UTC](https://discuss.elastic.co/t/unable-to-create-apikey-in-apmserver-after-installing-7-9/246659/14 "2020-09-23T05:48:56Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.