Connection Refused Error while sending metrics from APM-agent [JAVA]

[Parth (Parth Parmar)]

I am trying to implement APM stack with existing ElasticSearch & Kibana components.

I had deployed a dockerized APM Server on my http://localhost:8200 which is successfully talking to Elasticsearch. [I confirmed that it is working from Kibana UI & also by curl-ing it]

I am now trying to deploy a sample application on tomcat [which is a docker image]. My docker container runs successfully on http://localhost:8080 & my docker file looks like as below:

// FROM tomcat:8.0-alpine
// LABEL maintainer="parth.parmar@aeris.net"
// ADD sample.war /usr/local/tomcat/webapps/
// ADD elastic-apm-agent-1.7.0.jar /usr/local/tomcat/
// ADD setenv.sh /usr/local/tomcat/bin/
// RUN chmod +x /usr/local/tomcat/bin/setenv.sh
// EXPOSE 8080
// CMD ["catalina.sh", "run"]

My setenv.sh file looks like as below:

// export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/usr/local/tomcat/elastic-apm-agent-1.7.0.jar"
// export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=my-apm-agent-test"
// export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.application_packages=org.****"
// export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_urls=http://localhost:8200"

The APM agent seems to have been attached correctly but somehow it is not sending the metrics to APM -server. I am getting the following error:

// ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
// INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

I am not sure if I am doing something wrong. Can I please get some help with this?

[Eyal_Koren]

Hi and welcome to the forum!

Did you verify that the APM server is reachable from within the tomcat container through this URL, eg curl localhost:8200?

[felixbarny (Felix Barnsteiner)]

It seems like your application container can't talk to the APM Server container by using localhost. Make sure they both are in the same network and to configure the container name as the host name. For example, add the environment variable ELASTIC_APM_SERVER_URL=http://apm-server:8200, assuming the container name of the APM Server is apm-server.

When using docker, I'd advise to set the configuration via environment variables as opposed to add system properties to the CATALINA_OPS environment variable. By doing that, you can use the image with different configuration settings by specifying the environment variables when starting docker or by configuring in a docker-compose.yml file.

[Parth (Parth Parmar)]

Thanks a lot Eyal & Felix for your quick response. I had later figured out that there was some issue going on with my docker network interface because of which it was not letting apm-agent talk to apm-server. I have now resolved the network interface issue after which it is working fine.

Also, thanks for the recommendation Felix. I would definitely like to go with your idea of replacing system properties with docker environment variable for configuration easiness.