If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 
TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.
TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.
Kibana version: 8.2
Elasticsearch version: 8.2
APM Server version: 8.2
APM Agent language and version: nodejs
Browser version:
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Installed APM using Fleet server and elastic agent
Fresh install or upgraded from other version?
New install
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
Not using load balancer
Not change index pattern
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have configured APM setup on server side with following details :
./elastic-agent install --url=https://gnbsx20637.xx.yy.com:8220 --fleet-server-es=https://gnbsx20637.xx.yy.com:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjAwNDQ5MzMwNzA6X1ZjcmVoOGNTUGVuVkdPZG8tdFRYZw --fleet-server-policy=fsp3 --certificate-authorities=/data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/ca/ca.crt --fleet-server-es-ca=/data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/elasticsearch-ca.pem --fleet-server-cert=/data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/fleet-server/fleet-server.crt --fleet-server-cert-key=/data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/fleet-server/fleet-server.key
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
{"log.level":"info","@timestamp":"2022-08-09T13:56:51.653+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":783},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-09T13:56:53.654+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":764},"message":"Fleet Server - Running on policy with Fleet Server integration: fsp3; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-09T13:56:54.226+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Starting enrollment to URL: https://gnbsx20637.xx.yy.com:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-09T13:56:55.006+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":261},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
however in the apm server log file : /opt/Elastic/Agent/data/elastic-agent-b9a28a/logs/default/apm-server-20220810-5.ndjson, it is showing following error :
*{"log.level":"error","@timestamp":"2022-08-10T05:57:26.444+0200","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed b*
*y unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}*
The fleet.yml is as follows :
*agent:
id: 2be2c3e4-6c5c-4233-80a5-d0c27649ed19
monitoring.http:
enabled: false
host: ""
port: 6791
buffer: null
fleet:
enabled: true
access_api_key: Ry1Pamc0SUJGV0YzanFETzNYelI6UkNDNjVUOUVUYzJxakNDRVlOTXMtdw==
protocol: https
host: gnbsx20637.xx.yy.com:8220
ssl:
verification_mode: full
certificate_authorities:
- /data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/ca/ca.crt
renegotiation: never
timeout: 10m0s
proxy_disable: true
reporting:
threshold: 10000
check_frequency_sec: 30
agent:
id: ""
server:
policy:
id: fsp3
output:
elasticsearch:
protocol: https
hosts:
- gnbsx20637.xx.yy.com:9200
service_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjAwMzg4NjU1Mjc6YnRjOHQ2Mm1RVUNINE9WcjlIcFRFZw
ssl:
verification_mode: full
certificate_authorities:
- /data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/elasticsearch-ca.pem
renegotiation: never
proxy_disable: false
proxy_headers: {}
host: 0.0.0.0
port: 8220
internal_port: 8221
ssl:
verification_mode: full
certificate: /data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/fleet-server/fleet-server.crt
key: /data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/fleet-server/fleet-server.key
renegotiation: never*
I tried connecting the elasticsearch server without apm using the certificate and it is not working without username and passwd :
*curl --cacert/data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/elasticsearch-ca.pem https://gnbsx20637.xx.yy.com:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}-bash-4.2$ client_loop: send disconnect: Connection reset*
When curl run with username and passwd, it works :
*-bash-4.2$ curl --user elastic --pass xxxx --cacert /data/essw/elastic-agent-8.2.0-linux-x86_64/fleet-secure/elasticsearch-ca.pem https://gnbsx20637.xx.yy.com:9200
Enter host password for user 'elastic':
{
"name" : "node-3-gnbsx20637",
"cluster_name" : "cad-elasticsearch-qa",
"cluster_uuid" : "HW8-XdM1Rgig3Mxa1b85Mw",
"version" : {
"number" : "8.2.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5",
"build_date" : "2022-04-20T10:35:10.180408517Z",
"build_snapshot" : false,
"lucene_version" : "9.1.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
-bash-4.2$*
Steps to reproduce:
1.
2.
3.
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
Can you please help understanding where the issue is?
The elasticsearch-ca.pem certificate is the one that my kibana used to connect to es and was generated using : ./bin/elasticsearch-certutil http
unzip ../elasticsearch-ssl-http.zip
Archive: ../elasticsearch-ssl-http.zip
creating: elasticsearch/
creating: elasticsearch/node-1-gnbsx20635/
inflating: elasticsearch/node-1-gnbsx20635/README.txt
inflating: elasticsearch/node-1-gnbsx20635/http.p12
inflating: elasticsearch/node-1-gnbsx20635/sample-elasticsearch.yml
creating: elasticsearch/node-2-gnbsx20636/
inflating: elasticsearch/node-2-gnbsx20636/README.txt
inflating: elasticsearch/node-2-gnbsx206036/http.p12
inflating: elasticsearch/node-2-gnbsx20636/sample-elasticsearch.yml
creating: elasticsearch/node-3-gnbsx20637/
inflating: elasticsearch/node-3-gnbsx20637/README.txt
inflating: elasticsearch/node-3-gnbsx20637/http.p12
inflating: elasticsearch/node-3-gnbsx20637/sample-elasticsearch.yml
creating: kibana/
inflating: kibana/README.txt
inflating: kibana/elasticsearch-ca.pem
inflating: kibana/sample-kibana.yml
I am using self signed certificates for on-prem setup here.
