Java APM agent can't connect to APM server

[majarianil (Anil kumar)]

Kibana version: 7.12.0

Elasticsearch version: 7.12.0

APM Server version: 7.12.0

APM Agent language and version: Java - openjdk:17.0.2

Browser version: Google chrome - 05.0.5195.127

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker image

Fresh install or upgraded from other version? Fresh

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I have setup Elasticsearch locally and configuring my sample "app" to APM server through apm-java-agent. But i'm getting this error "Elastic APM server http://localhost:8200/ is not available (Connection refused)". And i have looked into the elasticsearch troubleshooting docs but not able to solve this issue. Could anyone has idea about this issue!!!

Provide logs and/or server output (if relevant):

 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'helloworld-service' (source: Java System Properties)  
 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://localhost:8200' (source: Java System Properties)
 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.example' (source: Java System Properties) 
 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker Elastic APM server http://localhost:8200/ is not available (Connection refused)
 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state 
 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused

docker-compose.yml:

version: "3.7"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: elasticsearch
    restart: always
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - elasticsearch-data-volume:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.12.0
    restart: always
    environment:
      SERVER_NAME: kibana
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
  apm-server:
    image: docker.elastic.co/apm/apm-server:7.12.0
    container_name: apm-server
    restart: always
    cap_add: [ "CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID" ]
    cap_drop: [ "ALL" ]
    command: >
      apm-server -e
        -E apm-server.rum.enabled=true
        -E setup.apm-server.host=0.0.0.0:8200
        -E setup.kibana.host=kibana:5601
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
    environment:
      SERVER_NAME: apm-server
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "8200:8200"
    depends_on:
      - elasticsearch
      - kibana

volumes:
  elasticsearch-data-volume:
    driver: local

Dockerfile (for app which I'm using apm agent with):

FROM openjdk:17.0.2

WORKDIR /app

COPY ./target/*.jar /app/

COPY --from=docker.elastic.co/observability/apm-agent-java:1.34.0 /usr/agent/elastic-apm-agent.jar /tmp/elastic-apm-agent.jar

EXPOSE 8080

#set the startup command to execute the jar
ENTRYPOINT ["/bin/sh", "-c", "java -javaagent:/tmp/elastic-apm-agent.jar -Delastic.apm.service_name=helloworld-service -Delastic.apm.application_packages=com.example -Delastic.apm.server_urls=http://localhost:8200 -jar /app/helloworld-*.jar"]
#INFO: i have also tried with the apm_server_url=http://apm-server:8200  where, apm-server is my apm server container name. But gettitng same error

[stephenb (Stephen Brown)]

I think it is because your hello world app is running in a separate docker container ... in that container localhost does not point to the localhost of the container that the apm server is running in. This is a docker networking issue.... remember every container is its own "host"

I suspect if you just ran your hello world from the actual host command line outside of docker it would probably work

And or read something like this

and change to something like this...

ENTRYPOINT ["/bin/sh", "-c", "java -javaagent:/tmp/elastic-apm-agent.jar -Delastic.apm.service_name=helloworld-service -Delastic.apm.application_packages=com.example -Delastic.apm.server_urls=http://host.docker.internal:8200 -jar /app/helloworld-*.jar"]

[majarianil (Anil kumar)]

yes, you are right it is because I'm running the app in a seperate container. Now i have changed the apm server url to (http://host.docker.internal:8200) and it is working.

Thank you for the suggestions and the solution