Docker Compose Tutorial - Connection to APM Server timed out

Elastic Observability
1

I'm trying to follow this simple guide to get APM up and working with docker-compose

I was able to do all the steps. My only issue is that the python app container gives the error:

docker logs es-cluster-webapp-1
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'
Failed to submit message: 'Connection to APM Server timed out (url: http://fleet-server:8200/intake/v2/events, timeout: 5 seconds)'

Everything else about this tutorial works, the agent is submitting mem+cpu information and showing metric, the fleet server says healthy etc... Just my web app container can't connect to the fleet server. I tried restarting the container, I tried restarting the entire docker-compose, but it is still the same result.

Where I can I go to get information on why the webapp container can't connect to APM? Or even better, what did I need to do to make the python app submit data to the APM server?

Just to summarize, these are the steps I followed from the guide

git clone  https://github.com/elkninja/elastic-stack-docker-part-two.git
cd elastic-stack-docker-part-two
docker-compose up --build -d
docker cp es-cluster-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.
openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g
cat /tmp/ca.crt

Then I pasted the following information into Kibana's Fleet Server Settings (as instructed by the tutorial)

HOSTS:
https://es01:9200/

FINGERPRINT:
DBF32B7752D6C345BA087FA55A47BF6D35CFEACBFC1577E5D8BDF77C6AF0CA2C

ADVANCE YML
ssl:
  certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDSjCCAjKgAwIBAgIVAP9hxK7zF+B1IgZJoQAPHxAVetPuMA0GCSqGSIb3DQEB
    CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
    ZXJhdGVkIENBMB4XDTIzMTEyMDIxNTczMFoXDTI2MTExOTIxNTczMFowNDEyMDAG
    A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
    ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCduXFAa+/dkXiAH3HGQqiI
    RfHtFqx7g8yX0/L221XYb8wOP4vhBUIELBbHCyezcjbjCE/n9Ipi1AFdm+yU4LOG
    ZHyipmMRo4aU7lrTJbDHwbkOF4DPAuV+hBZyKTA2a+s3cOq403+CZa7LRdkH9LMf
    7dGpS4NqpduVM9pT2z++K9X03xfLT0Z5341iLBHLrjbolK7drH83uQ+/q2cDupFM
    eOJsu+zfkuZWRGyGbMvddW4uRgWkzSUJ8rJ8FezyNl6gb+6mFHVdxNKRbFM91UxO
    kWy8jygjTHwcJSfPMusIc5TUflBaZn9NT74JCAuegwR78KDCP4W+csyzQCLfrs+L
    AgMBAAGjUzBRMB0GA1UdDgQWBBS+O9yqGbL9ms2z2roeSx30WzKd+DAfBgNVHSME
    GDAWgBS+O9yqGbL9ms2z2roeSx30WzKd+DAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
    SIb3DQEBCwUAA4IBAQBr2e5Qv4uTeewTMR96JGobBkxq33WcfsKp+P5NbUcSigj/
    lYgLwDgKPO7j9VkBWTpVT2BoTRKds92CDsE5nPwgj2FPfd+nwuWSMz2xPTtIWqtw
    9SiB8jgGZWdlOFcAoD7DyHXUParo28Z20fEEQ2lI//0YUfQuva0gJbFnwbPJ14BK
    kwBrnZLXKK9bngBfMEU9yPbOAl9zRv39YvAJXEDn0NXqYzyRPk0ncp3nopXOqXiB
    2W8XKHruHhAlkA87x4Hjr6F5HdjK5VnFHuV2ZRteltISjnZzULc66JgRVOI6Rifa
    A8SAhlyt2KWMs+dgPS6omYkfMXN3UPSyXBqKKCEv
    -----END CERTIFICATE-----

I pressed save and deploy. Then I see metrics and healthy status for the elastic agent. But no APM results appear in my Kibana, and when I do the docker logs es-cluster-webapp-1, I just see the time out issue.

What did I do wrong? Or how do I get more information on the timeout issue?

Here is a 3 min video of me doing all of these steps:

What did I miss?

2

Hi @learningelastic,

Welcome back! I've pinged the author to see if he has any ideas.

But I would recommend checking the connection settings and also double check if any messages are coming through from the Python app in case it's down to load rather than a straight block (see the troubleshooting guide here).

Hope that helps!

3

Hey @learningelastic!

First, thank you for such a great explanation and even a video recreating your problem! That makes discussion much easier for sure.

I'm going to wipe out my environment and rebuild using your steps to try recreate the issue myself.

if you access http://localhost:8000 from your browser - does the sample app load?

if you're able to interact with the web app, do any additional logs show up on the container?

1 Like
4

Ooooooh...how stupid of me . Yes, I need to actually visit the website at http://localhost-or-ip:8000/ and interact with the website first. I was just under the wrong impression that if I started the container, the python app would automatically broadcast some on-load event data to APM.

So yes, when I visit http://localhost-or-ip:8000/ and click around, data is actually going to my ELK stack.

Thanks!

5

Oh! Great! Glad its working.
I wasn't able to recreate the problem and your steps worked perfectly.
I think the error you saw was a residual because when the container first starts up, it wont be able to connect to the fleet-server and throws the error.