Issue with APM monitoring for docker containers

[calshankar (Shankar)]

I was able to successfully setup the EFK stack for docker container on a single host but i'm unable to get the agent from the app to communicate with the APM. I looked into the documentation & added the npm module & init config on top of the index.js file but with no luck.

Basically the communication btw app agent module & APM is broken thought the APM service is listening on port 8200

My compose config along with app code that brings up the stack @ https://github.com/calshankar/EFK-stack/blob/master/docker-compose.yml

since all containers are communicating on locahost, i thought it was easy to setup. Seems like config issue that i'm not able to trace

Could you point me to the source code of any demo docker nodejs app that I can cross reference to weed out the issue?

Also is there any issue in the way i bring up the stack using compose file? Are there any recommendation that i can take a look at

Here is the line from index.js

var apm = require('elastic-apm-node').start({
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: 'app-boron',
    // Use if APM Server requires a token
    secretToken: '',
    // Set custom APM Server URL (default: http://localhost:8200)
    serverUrl: 'http://127.0.0.1:8200'
})

const express = require('express'),
  mongodb = require('mongodb'),
  app = express(),
  bodyParser = require('body-parser'),
  url = process.env.MONGODB_URI || 'mongodb://localhost:27017/board'

[gil (Gil Raphaelli)]

@calshankar Thanks for the detailed report, I think you're pretty close.

The first issue is that apm-server binds to localhost:8200 by default, which doesn't work nicely in a container - can you add -E apm-server.host=0.0.0.0:8200 on https://github.com/calshankar/EFK-stack/blob/1f9559c00f0df9a710b8dfa5d74fece3b2c4d14b/docker-compose.yml#L151?

The second issue is helping the node app running in its container find the apm-server. For that, update serverUrl in index.js to point to http://apm_server:8200.

Please check back to let us know how it goes, good luck!

[calshankar (Shankar)]

Thank a ton for your response. I'm still not able to setup the connection btw the app(node-agent) & apm server though i got the apm server to communicate via curl, which indicates to me that apm server is open for communication

curl http://localhost:8200/v1/transactions
{"error":"only POST requests are supported"}

docker ps | grep -i apm

7f8bb2751d64 docker.elastic.co/apm/apm-server:6.2.4 "/usr/local/bin/do..." 5 minutes ago Up 5 minutes 0.0.0.0:8200->8200/tcp apm_server

I tried the changing the the IP to host ip from localhost in index.js ((https://github.com/calshankar/EFK-stack/blob/master/code/index.js) to no avail. All the elastic docker component are of version 6.2.4

Could you point me to the source code of any demo docker nodejs app that I can cross reference to weed out the issue with the app config? If you already h'v any demo docker app which works without any issue with apm then i can try that out

[gil (Gil Raphaelli)]

I'm not aware of a public project that sets things up as you have here.

I'm fairly certain that an update to serverUrl in index.js to point to http://apm_server:8200 will resolve the issue.

The issue is that the apm-server isn't on localhost relative to your node app. You can confirm this by running curl within your node application container.

[calshankar (Shankar)]

Thanks a ton @gil . Completely understand network namespace when the url is fired from inside of the app container. I actually tried with both the serverUrl: 'http://apm_server:8200' & serverUrl: 'http://host-ip:8200' yesterday in vain, realizing the problem could be elsewhere.

Then i realised that the issue could be with the npm package which might not have installed properly in the global name-space. So, i rebuilt the docker image of the app with all the dependencies instead of installing packing @ runtime. Voila..! i got the apm instrumentation working.

I'll share the repo so that anyone could try it out. I'm trying to get metric beat data for mongo DB to show up & once done, i'll share the repo for a very simple use case

@ruflin @simianhacker Is their anything specific to setup inside of mongodb container metric beat dashboard to show up data? I can see some info on packetbeat dashboard for mongo

Another issue that I have is that, I can see only containers of metric beat in Containers Overview Dashboard not all the containers running on the host, though the Docker container dashboard shows up running containers along with the instrumented app.

I guess my compose config @ https://github.com/calshankar/EFK-stack/blob/b429b7f0021aee7c6709a0cbd548dcbbea14c7ea/docker-compose.yml#L119 has necessary volumes of the host os mounted as 'ro' mode but still i don't see my app containers & other containers running on the host. i'm on openSuse linux kernel 4.13.9

Is there any config for metricbeat to instrument all the containers on the host which is missing in my compose file?