APM docker-compose connectivity problems

Elastic Observability APM
1

Hello everyone,

I'm encountering a very strange problem at the moment with APM server. I'm trying to integrate it into our docker-compose setup, version of all components is 7.9.0. When the containers start up, I get this error:

Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200))
and
504 Gateway Timeout

The config is pretty much default, some others have this issue with 401 due to missing credentials, that's not the problem in my case.

I also tried pinging elasticsearch from within the apmserver container, works as expected, so it shouldn't be related to the docker networking. Kibana in the same setup works fine with elasticsearch. Any ideas, what might cause this or what I can try to get it working? I've been reading the documentation for an hour now and can't find any errors in the configuration.

My docker-compose.yml:

 elasticsearch:
   image: ${DOCKER_REGISTRY-}elasticsearch:7.9.0
     volumes:
       - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
       - ../config/elasticsearch/data/:/usr/share/elasticsearch/data
       ports:
         - "9200:9200"
         - "9300:9300" 

  apmserver:
    image: docker.elastic.co/apm/apm-server:7.9.0
    volumes:
      - ../config/apmserver/apm-server.yml:/usr/share/apm-server/apm-server.yml
    ports:
      - "8200:8200"
2

Have you tried using curl http://elasticsearch:9200/ to query Elasticsearch's health?

I noticed in your config you've got "../config" in two places, and "./config" in one place. Do you really have two config directories, or is one of those a typo?

3

Hey Andrew.

The first ./config is actually a ../config, must've been lost when I copied the file. I just tried hitting elastic-search health api from inside the apm-container with curl, this yields the same error from the apm-server logs. From outside of docker network on my localhost, I can reach the health API of ES without issues via localhost:9200....This is the html that curl returns inside the apm-container:

504 Gateway Timeout

Gateway Timeout

The requested URL couldn't be resolved

Same thing from kibana-container, but kibana manages to connect to elasticsearch. My ES-config looks like this:

network.host: 0.0.0.0
http.host: 0.0.0.0
http.port: 9200

discovery.type: single-node

4

Update:

I was able to resolve the issue. My containers were connecting via our corporate HTTP-proxy. Had to include elasticsearch to the no_proxy list. Running curl in verbose mode finally gave me the solution, so thanks Andrew for pointing that out.