Apm server as kubernetes daemonset?

[ethrbunny]

I want to start deploying pods with elastic apm monitoring. To this end I need to (?) get the apm server deployed as a daemonset so it will be available on every k8 node.

So far no luck from a test java app. I checked the test pod and the process looks like this:

java -Dlogback.configurationFile=./logback.xml -jar /mytestapp-all.jar -javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=mytestapp -Delastic.apm.application_packages=com.testapp -Delastic.apm.server_url=http://localhost:8200

Here's the deployment file Im using:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  namespace: logging
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
      frontend:
        enabled: false
    setup.template.settings:
      index:
        number_of_shards: 1
        codec: best_compression
    output.logstash:
      hosts: ["10.95.96.75:5044"]
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: apm-server
  namespace: logging
  labels:
    k8s-app: apm-server
spec:
  # this replicas value is default
  # modify it according to your case
  #replicas: 1
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:6.5.3
        ports:
        - containerPort: 8200
          name: apm-port
        volumeMounts:
        - name: apm-server-config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config

Im missing a piece somewhere.

[roncohen]

Hi there!

Thank you for your interest in Elastic APM.

As far as I can tell, your daemonset will not be available through "localhost".
You might need to use the DownwardAPI to find the host ip and use that in the configuration of the agent. This article seems to describe your scenario: https://bravopooper.wordpress.com/2017/07/17/communicating-with-local-daemon-set-pods-in-kubernetes/

Then, setting the APM Server URL to using something like this should work:
-Delastic.apm.server_url=http://$HOST_IP:8200

There's also this unofficial helm chart that might be interesting: https://github.com/helm/charts/pull/6058

I should mention that we recently launched official Helm charts for Elasticsearch. I hope this will also extend to APM Server in the future.

Let me know how it goes!

/Ron

[ethrbunny]

For '-Delastic.apm.server_url=' Ive tried:

• localhost
• 127.0.0.1
• container ip

all to no avail.

Im thinking that the apm-server config isn't correct. 'nmap' for all of the above IP addresses shows a 'closed' port (8200).

Note - Ive tried deploying the daemonset with all of the above values for 'apm-server/host' (also '0.0.0.0')