APM agent can't connect to APM

Kibana version: 7.7.0

Elasticsearch version: 7.7.0

APM Server version: 7.7.0

APM Agent language and version: Java - openjdk:8-jre-alpine

Browser version: Firefox 88.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker image

Fresh install or upgraded from other version? Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I cannot configure APM due to error "No data has been received from agents yet". I found in troubleshooting that my error is ithat agent can't connect to Elastic APM Server (Troubleshooting | APM Java Agent Reference [1.x] | Elastic) - logs below.
However, I don't know what next steps should be to solve this issue.

Provide logs and/or server output (if relevant):

2021-04-29 17:45:27,012 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2021-04-29 17:45:27,177 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.23.0 as api-gateway on Java 1.8.0_212 Runtime version: 1.8.0_212-b04 VM version: 25.212-b04 (IcedTea) Linux 4.19.128-microsoft-standard
2021-04-29 17:45:27,177 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-javaagent:/agent.jar, -Delastic.apm.service_name=api-gateway, -Delastic.apm.server_url=http://127.0.0.1:8200, -Delastic.apm.application_packages=pl.agh.apigateway]
2021-04-29 17:45:27,178 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'api-gateway' (source: Java System Properties)
2021-04-29 17:45:27,178 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'http://127.0.0.1:8200' (source: Java System Properties)
2021-04-29 17:45:27,178 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'pl.agh.apigateway' (source: Java System Properties)
2021-04-29 17:45:30,727 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-04-29 17:45:30,782 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server http://127.0.0.1:8200/ is not available (Connection refused (Connection refused))
2021-04-29 17:45:30,782 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused (Connection refused)

docker-compose.yml:

version: '3.8'

services:
  api-gateway:
    container_name: "api-gateway"
    hostname: "api-gateway"
    build: ../../api-gateway/target
    ports:
      - "8080:8080"
    expose:
      - "8080"
    depends_on:
      - consul
      - logstash
      - kibana
      - consul2
      - consul3
    networks:
      - backend

  service-calculator:
    container_name: "service-calculator"
    hostname: "service-calculator"
    build: ../../service-calculator/target
    ports:
      - "5555:5555"
    expose:
      - "5555"
    depends_on:
      - api-gateway
      - consul
      - logstash
      - kibana
      - consul2
      - consul3
    networks:
      - backend

  service-database:
    container_name: "service-database"
    hostname: "service-database"
    build: ../../service-database/target
    ports:
      - "6666:6666"
    expose:
      - "6666"
    depends_on:
      - api-gateway
      - consul
      - logstash
      - kibana
      - consul2
      - consul3
    networks:
      - backend

  service-history:
    container_name: "service-history"
    hostname: "service-history"
    build: ../../service-history/target
    ports:
      - "7777:7777"
    expose:
      - "7777"
    depends_on:
      - api-gateway
      - service-history-db
      - consul
      - logstash
      - kibana
      - consul2
      - consul3
    links:
      - service-history-db
    networks:
      - backend

  service-history-db:
    container_name: "service-history-db"
    hostname: "service-history-db"
    image: "mongo:3.7"
    environment:
      - MONGO_DATA_DIR=/data/db
      - MONGO_LOG_DIR=/dev/null
    volumes:
      - mongodata:/data/db
    ports:
      - "8888:8888"
    expose:
      - "8888"
    command: mongod --port 8888
    networks:
      - backend

  consul:
    image: consul:1.7.3
    container_name: "consul"
    hostname: "consul"
    command: consul agent -server -client 0.0.0.0 -ui -bootstrap-expect=3 -data-dir=/consul/data -retry-join=consul2 -retry-join=consul3 -datacenter=blr
    ports:
      - "8500:8500"
      - "8600:8600"
    networks:
      - backend
  consul2:
    image: consul:1.7.3
    container_name: "consul-2"
    hostname: "consul2"
    expose:
      - "8500"
      - "8600"
    command: consul agent -server -data-dir=/consul/data -retry-join=consul -retry-join=consul3 -datacenter=blr
    links:
      - consul
    networks:
      - backend
  consul3:
    image: consul:1.7.3
    container_name: "consul-3"
    hostname: "consul3"
    expose:
      - "8500"
      - "8600"
    command: consul agent -server -data-dir=/consul/data -retry-join=consul -retry-join=consul2 -datacenter=blr
    links:
      - consul
      - consul2
    networks:
      - backend

  # ELK STACK
  elasticsearch:
    image: elasticsearch:7.7.0
    container_name: elasticsearch
    hostname: elasticsearch
    environment:
      - "cluster.name=elasticsearch"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=127.0.0.1"
      - "discovery.type=single-node"          
      - "http.host=0.0.0.0"
      - "transport.host=127.0.0.1"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "9200:9200"
      - "9300:9300"
    expose:
      - "9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - backend

  logstash:
    image: logstash:7.7.0
    container_name: logstash
    hostname: logstash
    ports:
      - "5000:5000"
    expose:
      - "5000"
    volumes:
      - /Users/Shared/data/tools/logstash/:/logstash-container/
    command: >
      logstash --debug -e 'input { tcp { port => 5000 codec => json { charset => "UTF-8" } } } output { elasticsearch { hosts => "elasticsearch:9200" } }'
    depends_on:
      - elasticsearch
    networks:
      - backend

  kibana:
    image: kibana:7.7.0
    container_name: kibana
    hostname: kibana
    ports:
      - "5601:5601"
    expose:
      - "5601"
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    links:
      - elasticsearch
    networks:
      - backend

  apm-server:
    image: store/elastic/apm-server:7.7.0
    container_name: apm-server
    depends_on:
      - elasticsearch
      - kibana
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    ports:
    - 8200:8200
    networks:
    - backend
    command: >
       apm-server -e
         -E apm-server.rum.enabled=true
         -E setup.kibana.host=http://kibana:5601
         -E setup.template.settings.index.number_of_replicas=0
         -E apm-server.kibana.enabled=true
         -E apm-server.kibana.host=http://kibana:5601
         -E output.elasticsearch.host=http://elasticsearch:9200
    healthcheck:
      interval: 10s
      retries: 12
      test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

networks:
  backend:
    driver: bridge

volumes:
  mongodata:
  logstash-container:
  esdata:

Dockerfile (for api-gateway which I'm using apm agent with):

# we will use openjdk 8 with alpine as it is a very small linux distro
FROM openjdk:8-jre-alpine

EXPOSE 8080

# copy the packaged jar file into our docker image
ADD api-gateway-0.0.1-SNAPSHOT.jar app.jar
ADD elastic-apm-agent-1.23.0.jar agent.jar

# set the startup command to execute the jar
CMD ["java", "-javaagent:/agent.jar", "-Delastic.apm.service_name=api-gateway*", "-Delastic.apm.server_url=http://127.0.0.1:8200", "-Delastic.apm.application_packages=pl.agh.apigateway", "-jar", "/app.jar"]
1 Like

According to provided docker-compose, the hostname of APM Server is apm-server (the same as container name). As APM Server is located in different container, while referring to from api-gatway to localhost, you refer to the.. api-gatway itself. In order to connect to apm-server you have to specify apm-server's hostname (which thanks to DNS will be translated into the real container IP).

So, to solve your issue, just change parameter value of -Delastic.apm.server_url from http://127.0.0.1:8200 to http://apm-server:8200.

2 Likes

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.