Elastic APM agent not pushing data to APM server from docker but pushing data outside of docker

[Rishav_Kr]

APM Agent language and version: 6.18.0 with Python (FastAPI)

Issue :
When executing the APM agent with Python (FastAPI) outside docker I am able to view the API transactions in Kibana, however, when I run the FastAPI server from docker, I can't see the transactions in Kibana.

Network checks:
From inside the docker, I have checked using telnet and APM server is accessible. Telnet connected to port 8200, 5044. If I disable the VPN I can see error log stating server url is not accessible, where as with VPN there is no error log wrt to connecting to APM server.
From outside docker, things are working as expected.

Logs :
Agent has status code 202 in logs. Transaction ID is null from the instance inside docker and not from outside docker.
I am unable to figure out why transaction_id, trace_id is null?

(Inside docker) :
{"elasticapm_transaction_id": null, "elasticapm_trace_id": null, "elasticapm_span_id": null, "elasticapm_service_name": "Samiksha", "elasticapm_service_environment": "test", "elasticapm_labels": {"transaction.id": null, "trace.id": null, "span.id": null, "service.name": "Samiksha", "service.environment": "test"}

{"@timestamp":"2023-09-15T06:07:58.112Z","log.level":"debug","message":"Sent request, url=http://xxxxxx:8200/intake/v2/events size=0.48kb **status=202**","ecs":{"version":"1.6.0"},"log":{"logger":"elasticapm.transport.http","origin":{"file":{"line":96,"name":"http.py"},"function":"send"},"original":"Sent request, url=http://xxxxxx:8200/intake/v2/events size=0.48kb status=202"},"process":{"name":"MainProcess","pid":36578,"thread":{"id":140317223352064,"name":"eapm event processor thread"}},"service":{"environment":"test","name":"Samiksha"}}

(Outside docker) :
{"elasticapm_transaction_id": "a8156e7df65e0100", "elasticapm_trace_id": "0a11c5db87c1314ced4ade9862b24f37", "elasticapm_span_id": null, "elasticapm_service_name": "Samiksha", "elasticapm_service_environment": "test", "elasticapm_labels": {"transaction.id": "a8156e7df65e0100", "trace.id": "0a11c5db87c1314ced4ade9862b24f37", "span.id": null, "service.name": "Samiksha", "service.environment": "test"}

[basepi (Colton Myers)]

It appears that the transaction is not being activated for some reason inside of your docker container. I'm guessing that the 202 request you're seeing is for metrics or something else -- we don't send null transactions to the APM server.

How are you setting up the agent? Are you configuring via environment variables or in the code itself?

[Rishav_Kr]

Thanks for responding @basepi .
I am initializing the agent in the code itself.
Here is the snippet below.

from starlette.applications import Starlette
from elasticapm.contrib.starlette import make_apm_client, ElasticAPM

apm = make_apm_client({
    'SERVICE_NAME': 'XXXXXXXX',
    'SECRET_TOKEN': 'XXXXXXXXXXXXXXXX',
    'SERVER_URL': 'http://XXXXXXXX:8200',
    'VERIFY_SERVER_CERT': False,
    'ENVIRONMENT': 'production',
    'CAPTURE_BODY': 'errors',
})

app = FastAPI()
app.add_middleware(ElasticAPM, client=apm)

FastAPI version : fastapi==0.61.0

[basepi (Colton Myers)]

Do you happen do have any of these defined as environment variables? As noted in our configuration documentation, environment variables would override the inline configuration in your code.

In addition to potential environment variable pollution, can you think of anything else that is different about your docker setup? Could you perhaps share your Dockerfile? I'm at a loss for what could be causing this.

[Rishav_Kr]

@basepi None of the elastic apm configuration variables are used in the environment variables.
Here is the Dockerfile

# nvidia cuda image. See this --> https://github.com/NVIDIA/nvidia-docker
FROM nvidia/cuda:11.0.3-base-ubuntu18.04

#Set up environment
RUN apt-get update && apt-get install --no-install-recommends --no-install-suggests -y curl

RUN apt-get install -y locales
RUN apt-get update && apt-get -y install python3.8
RUN apt-get -y install python3-pip
RUN apt-get -y install poppler-utils
# open cv FIX
RUN apt-get update
RUN apt-get install 'ffmpeg'\
    'libsm6'\
    'libxext6'  -y
## Required for pyzbar
RUN apt-get install libzbar0 -y

# set the locale
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

COPY requirements.txt /
#Install all depnedency
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt  --default-timeout=100 --find-links=https://download.pytorch.org/whl/torch_stable.html --use-feature=2020-resolver

#Create working directory
WORKDIR /api
ADD . /api
EXPOSE 6600
CMD ["./startup.sh"]