Automatic APM agent cannot connect to apm server

Elastic Observability APM
1

Kibana version: 7.12

Elasticsearch version: 7.12

APM Server version: 7.12

APM Agent language and version: Java 1.22

Browser version: Chrome 86

Original install method (e.g. download page, yum, deb, from source, etc.) and version: source

Fresh install or upgraded from other version? fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
No
There are no SSL/TLS enabled and disabled the firewall too
SSL disabled in apm-server.yml too

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have ELK stack with Apm-server in one ec2 instance and installed java apm-agent for monitoring jenkins in another ec2 instance

I followed the automatic setup for apm-agent

Command used:

java -jar apm-agent-attach-standalone.jar --include '.*jenkins.*' --config service_name=jenkins-service  --config server_url=http://privateipofapmserver:8200

I can curl the apm-server instance and got 200 response

Steps to reproduce:

  1. Started Jenkins service
    2.Execute the command
    3.Check output

Errors in browser console (if relevant):

2021-03-30 03:52:59.425  INFO Attaching the Elastic APM agent to 7816 /usr/share/jenkins/jenkins.war -Djava.awt.headless=true with arguments {service_name=jenkins-service, server_url=http://x.x.x.x:8200}
2021-03-30 03:53:20.638  INFO Done
jenkins@ip-x.x.x.x:/etc/apmagents$ java -jar apm-agent-attach-standalone.jar --list
7816 /usr/share/jenkins/jenkins.war -Djava.awt.headless=true
24190 apm-agent-attach-standalone.jar

Provide logs and/or server output (if relevant):
1.No errors in apm-server log
2.Error in jenkins.log

2021-03-30 04:31:11,928 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server. Some details about SSL configurations corresponding the current connection are logged at INFO level.
2021-03-30 04:31:11,928 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: Connection refused (Connection refused)
2021-03-30 04:31:11,928 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2

Hi @Kevin_Christopher,

You should set the agent configuration log_level=debug to enable more detailed logging of the possible reasons the APM agent can't reach the server.

As the agent log will be more verbose, you might also set log_file to log to an external file so it won't pollute your jenkins logs.

See logging documentation reference for more details.

3

Hi @Sylvain_Juge ,

Thanks for the quick response.

Actually I repeated the steps for new jenkins server. It worked.
However I am not sure where to add the configuration log_level=debug and log_file.I tried with below command and file not created.

As you mentioned not to pollute the application log, your details will help

Can you explain it a bit more of where to setup this?

java -jar apm-agent-attach-standalone.jar --include '.*jenkins.*' --config service_name=jenkins-service --config server_url=http://x.x.x.x:8200 --config elastic.apm.log_file=/var/log/apm.log  --config log_level=debug
4

When using the --config option, you don't have the use the elastic.apm. prefix, thus if you remove it the following should work:

java -jar apm-agent-attach-standalone.jar --include '.*jenkins.*' --config service_name=jenkins-service --config server_url=http://x.x.x.x:8200 --config log_file=/var/log/apm.log  --config log_level=debug
5

@Sylvain_Juge
Thank you very much

6

@Sylvain_Juge

When I use the below command ,I am not getting any logs in new log file and jenkins.log too but data is received in apmserver and kibana. log levels are also not changing.

java -jar apm-agent-attach-standalone.jar --include '.*jenkins.*' --config service_name=jenkins-service --config server_url=http://x.x.x.x:8200 --config log_file=/var/log/apm.log  --config log_level=debug

response:

2021-04-05 05:43:05.089  INFO Attaching the Elastic APM agent to 27602 /usr/share/jenkins/jenkins.war -Djava.awt.headless=true with arguments {service_name=jenkins-service, server_url=http://x.x.x.x:8200}
2021-04-05 05:43:05.545  INFO Done

Note:
No errors in apm-server,jenkins,apm-agent

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.