APM Agent Status states "No data has been received from agents yet"

Elastic Observability APM
1

Hi,
I have deployed and configured APM which on clicking APM server status states as "You have correctly setup APM Server"

I am successfully running a Petclinic application using the below script:
sudo java
-javaagent:elastic-apm-agent.jar
-Delastic.apm.service_name=petclinic-java
-Delastic.apm.secret_token=
-Delastic.apm.application_packages=co.elastic.apm
-Delastic.apm.server_urls=https://:8200
-jar ./target/spring-petclinic-2.4.2.jar

However, it errors out with following connection refused:
2021-03-25 17:22:49.862 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2021-03-25 17:23:26.633 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused (Connection refused)
2021-03-25 17:23:26.634 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2021-03-25 17:23:59.075 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused (Connection refused)
2021-03-25 17:23:59.075 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2021-03-25 17:24:33.093 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused (Connection refused)
2021-03-25 17:24:33.094 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2021-03-25 17:25:03.908 [apm-remote-config-poller] ERROR co.elastic.apm.agent.report.HttpUtils - Exception when closing input stream of HttpURLConnection.
2021-03-25 17:25:03.909 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused (Connection refused)
2021-03-25 17:25:10.838 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused (Connection refused)

My apm-server.yml file (v 7.10.1) on RHEL7.0 has no changes except updates to the section, output.elasticsearch: in the file.

What do I need to setup or verify to find what may causing the error.
Is it required that the secret_token be provided in the config file and in the java script above,?

Please help

2

Hi @Stud21 ,

it seems that the both properties have invalid values, make sure that server_urls and secret_token are properly set.

server_urls should be a valid URL, here it seems that the host/ip part is missing.

3

I have entered entered the domain name in the apm.server.url as under:
-Delastic.apm.server_urls=https://stud21.com.au:8200 \

However, I am not sure how and what to entered as secret token.

Please advise

4

I have tried to enter in apm-server.yml as under:

Configure authorization via a common secret_token. By default it is disabled.

Agents include the token in the following format: Authorization: Bearer .

It is recommended to use an authorization token in combination with SSL enabled,

and save the token in the apm-server keystore.

secret_token: secret

The same was provided as in the java script as under:
-Delastic.apm.secret_token=secret

It still returns the same error.

5

In Kibana, go to Observability > APM, then click on the top-right Add Data button

Or just go to https://<host:port>/app/home#/tutorial/apm path where <host:port> is the hostname and port of your Kibana instance.

Then you'll have a secret token with setup instructions for each platform, while the instructions differ on every platform the token remains the same here.

6

I am not able to find any token as seen from the setup instructions below:
java -javaagent:/path/to/elastic-apm-agent-.jar
-Delastic.apm.service_name=my-application
-Delastic.apm.server_urls=http://localhost:8200
-Delastic.apm.secret_token=
-Delastic.apm.application_packages=org.example
-jar my-application.jar

Does it mean that I leave it blank?

7

The below command runs fine in linux as:
sudo java -javaagent:elastic-apm-agent-1.22.0.jar
-Delastic.apm.service_name=petclinic-java
-Delastic.apm.server_urls=https://localhost:8200
-Delastic.apm.secret_token=
-Delastic.apm.application_packages=co.elastic.apm.petclient
-jar spring-petclinic-2.4.2.jar

The petclinic application runs fine too and I can access it as normal.

However, clicking on Agent Status in APM still states as, No data has been received from agents yet.

Any clues?

8

Few things to check:

  • do you see anything captured in Kibana for this application ?
  • do you see anything in the apm agent logs ? By default it's included in the application standard output.
  • if you don't have any token, then you should just skip the property (not use an empty value)

The minimal configuration does not include secret_token, and it is not mandatory, however it's quite important as it allows to secure communication between apm agent and server. Depending on your apm-server setup it might be required or not. For example default apm-server configuration on Elastic Cloud instances require it, which might not be the case if you are using a local deployment.

9

Hi Sylvan,
in reply to your queries, see below:

  • do you see anything captured in Kibana for this application ?
    Response: None, In Discover for index pattern apm-* says " No results match your search criteria"
  • do you see anything in the apm agent logs ? By default it's included in the application standard output.
    Response: it says" There are no log messages to display." My logs is not working now.
  • if you don't have any token, then you should just skip the property (not use an empty value)
    Response: I've removed that condition line an it runs fine still and the Petclinic application is running.

In the end, Agent Status in APM still states as, No data has been received from agents yet.

10

When running the agent using, sudo java -javaagent:elastic-apm-agent-1.22.0.jar... I get the below error:
[elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: Unrecognized SSL message, plaintext connection?

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.