APM dashboard is not getting loaded

[Karthik_Medam]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version:
6.6.1
Elasticsearch version:
6.6.1
APM Server version:
1.6.1
APM Agent language and version:
latest
Browser version:
Version 79.0.3945.130
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
yum
Fresh install or upgraded from other version?
fresh install
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
We have provided Kibana loadbalancer
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.Getting "No services were found" under APM app.
2.
3.

Errors in browser console (if relevant):
command throws error: apm-server setup -E setup.kibana.host=https://10.8.16.20:5601 -e

2020-01-24T08:21:33.652Z ERROR instance/beat.go:911 Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get https://10.8.16.20:5601/api/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Response: .

Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get https://10.8.16.20:5601/api/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Response: .
Provide logs and/or server output (if relevant):

[jalvz (Juan Alvarez)]

Hi there!

I'm not sure I understand your problem completely, but a few things seem off track:

• Kibana dashboards with APM data became deprecated in 6.4, what you get instead is a specific APM tab in Kibana. That corresponds to the screenshot you are posting, but you don't need to pass setup.kibana.host to apm-server.

• The apm-server setup you are using simply creates the templates in Elasticsearch for APM. To have apm-server accept events you need to run apm-server -e. It will listen on port 8200 by default.

• The most important thing you need to do is configuring APM Server so it can talk to Elasticsearch: that is output.elasticsearch.hosts, output.elasticsearch.username and output.elasticsearch.password in apm-server.yml.
  Configure SSL/TLS too if needed. See https://www.elastic.co/guide/en/apm/server/6.6/elasticsearch-output.html.
  You can keep the index and indices setting as default.
  You can then test your connection running apm-server -e test output

• Then you need to configure your Elastic APM agent to point to APM Server, and use your app for a bit so it generates traffic data.

Finally, your version is a bit old, generally is a good idea to upgrade to a newer version, latest right now is 7.5.

I hope this can get you started!

/Juan

[jalvz (Juan Alvarez)]

Hello,

You are running apm-server and the agent in the same host; and you are pointing the agent to the apm-server and not to Elasticsearch directly, correct?
Can you share the result of running apm-server -e test output, and also the logs from both apm-server and the agent?

[Karthik_Medam]

Here are the agent logs and server logs.

[jalvz (Juan Alvarez)]

Ok,

So APM Server can connect to Elasticsearch, but it looks like the agent can't connect to APM Server. In the first screenshot there is a line saying "Elastic APM server https://...:8200 is not available (Connection refused)"

Just a guess: It might be related to SSL configuration? you need a private key and a certificate issued by a certification authority and specify the path to those files in apm-server.ssl.key and apm-server.ssl.certificate. Or, if you are using a self-signed cert, you need to disable verification in the agent (-Delastic.apm.verify_server_cert=false)

Also, for the next time: it is better to share logs as text than as screenshot: I can't search or copy from images