Docker-compose + fleet + RUM + APM on version>= 7.16

[jan499 (Jan)]

We have Elasticsearch, Kibana and APM running happily on version 7. However, we want to move to version 8, and because of that we first try to get a working version 7.17.

My kibana claims that the APM agent is running and the fleet service is running however the rum javascript receives status 0 when trying to connect to port 8200.
I do have enabled RUM and anonymous access in the APM agent settings.

So I wonder, which process is supposed to handle the 8200? is it the fleet agent, is it kibana, or is it somehow Elasticsearch server itself?

My docker compose looks like this:

services:
  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
    container_name: elastic
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9200"]
      timeout: 2s
      retries: 20
    ports:
      - "9200:9200"
    ulimits:
      nofile: # https://www.elastic.co/guide/en/elasticsearch/reference/current//docker.html#_increase_ulimits_for_nofile_and_nproc
        soft: 65536
        hard: 65536
    environment:
      - "discovery.type=single-node" # ontwikkelomgeving hoeft geen cluster te joinen
      - xpack.security.enabled=true
      - xpack.security.authc.api_key.enabled=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - path.repo=/usr/share/elasticsearch/backup
    volumes:
      - elastic_data:/usr/share/elasticsearch/data
      - elastic_backup:/usr/share/elasticsearch/backup
  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0
    volumes:
      - ./docker/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    environment:
      - "ELASTIC_USERNAME=kibana_system"
      - "ELASTIC_PASSWORD=......some_password that i generated during installation...."
    ports:
      - "5601:5601"
      - "8200:8200" # does not work. I also tried to add this port mapping to elastic-agent. but neither works....
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:7.17.0
    container_name: elastic-agent
    restart: always
    user: root # note, synthetic browser monitors require this set to `elastic-agent`
    ports:
      - "8220:8220"
    environment:
      - FLEET_ENROLLMENT_TOKEN=... a token that i found in Kibana......
      - FLEET_ENROLL=1
      - FLEET_URL=http://127.0.0.1:8220
      - FLEET_SERVER_ENABLE=true

[Alberto_Delgado]

Hi @jan499 ,

8200 is the default port for the APM server.

rum javascript receives status 0 when trying to connect to port 8200.

Which is the endpoint that RUM tries to request?

In the past I had a similar issue with the status 0 using dockers.

What I did is:

1. add the rule 127.0.0.1 apm-server in my /etc/hosts file.
2. set the config serverUrl to apm-server:8200 when initialising the RUM agent:

     elasticApm.init({
         serviceName: 'your service name',
         serverUrl: 'http://apm-server:8200',
    })

Let me know if this helps you

Thanks,
Alberto

[jan499 (Jan)]

the statuscode 0 was a side issue because i forgot to mention the protocol in the RUM initialisation.

But this doesnt solve my question.

When we add an APM server to our docker-compose (see config below), it works.
However. this APM server is "legacy" according to the documentation of 7.17 and 8 and I am under the impression (after reading the docs) that fleet should be used to somehow install an agent which does the thing the legacy server did before.

So i wonder, did i miss something in my docker-compose file, or can i just press some buttons in kibana to add a fleet process, and if so, which process is going to listen to 8200 and should be mapped in the docker-compose file.

Because although kibana claims that an APM server is "running" (there is a test button and it turns green if i press it), nothing will be logged and the client will throw exceptions unless the legacy apm server is also running.

screenshot of the test button about the APM server in kibana (it shows the green message even if the legacy APM server is removed from docker-compose):

legacy server docker compose fragment:

# legacy apm server. this is deprecated.
apm-server:
  image: docker.elastic.co/apm/apm-server:7.17.1
  cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
  cap_drop: ["ALL"]
  ports:
    - 8200:8200
  env_file:
    - .env.apm-server
  command: >
    apm-server -e
      -E apm-server.rum.enabled=true
      -E setup.kibana.host=kibana:5601
      -E setup.template.settings.index.number_of_replicas=0
      -E apm-server.kibana.enabled=true
      -E apm-server.kibana.host=kibana:5601
      -E apm-server.kibana.username=elastic
      -E apm-server.kibana.password=............
      -E output.elasticsearch.hosts=["elastic:9200"]
      -E output.elasticsearch.username="elastic"
      -E output.elasticsearch.password=...............
      -E apm-server.rum.allow.origins=.... some domains we serve....
      -E apm-server.rum.enabled=true
  healthcheck:
    interval: 10s
    retries: 12
    test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

[simitt (Silvia Mitter)]

Hi @jan499 ,
running APM Server managed by Elastic Agent and Fleet, you have to add the APM Integration (with any custom configuration settings) to the Agent Policy, to which the configured Elastic Agent is subscribed to. The Elastic Agent then receives this information from the Fleet Server component, and starts APM Server, listening on the configured port and URL.
Related to your docker setup this means that you need to expose the port 8200 on the elastic-agent.

[jan499 (Jan)]

Thanks Silvia,

i added the integration, but it doesnt work.
could it be that the policy of elastic agent isn't right?

because when i look at what runs it runs the default fleet server policy.

wheras apm seems to be added in the default policy:

does this also mean that i need 2 agents in my docker compose? or how do i tell the agent which policy it uses?

[simitt (Silvia Mitter)]

FLEET_SERVER_ENABLE: true tells the Elastic Agent to set up a Fleet Server, which triggers the enrollment to the Default Fleet Server Policy.

The FLEET_ENROLL, FLEET_URL and FLEET_ENROLLMENT_TOKEN should be configured if you set up an Elastic Agent that is supposed to start up in a non-Fleet Server mode, and instead should enroll through the already running Fleet Server.

Here is the recomendation I received from the Elastic Agent and Fleet Server developers:
For a non-production environment you could add the APM Integration to the Default Fleet Server Policy, and then there'd be one Elastic Agent that runs Fleet Server and APM Server.

For a production environment the recommendation is to run a dedicated container for the Fleet Server and another one for the Elastic Agent with any configured integrations.