How to get ECK APM to work with any type of agent at all!?!

[Kok_How_Teh]

var apm = require('elastic-apm-node').start({
  serviceName: 'my-apm',
  secretToken: '',
  // Set custom APM Server URL (default: http://127.0.0.1:8200)
  serverUrl: 'https://my-apm-http:8200',
  // Only activate the agent if it's running in production
  active: process.env.NODE_ENV === 'production',
  verifyServerCert: false,
  serverCaCertFile: '/etc/ca-certificates/apm.crt',

Errors:

[myapp-0 myapp] {"log.level":"trace","@timestamp":"2023-08-23T06:13:21.947Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"83fb5052fa3f5a4420824fabac133671","timeline":[[10002.293778,"completePart gzipStream",null],[10002.594401,"completePart intakeReq",null],[10003.133811,"completePart intakeRes","Unexpected APM Server response"]],"bytesWritten":731,"backoffDelayMs":15952.873427495506,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"Unexpected APM Server response","stack_trace":"Error: Unexpected APM Server response\n    at processIntakeErrorResponse (/node_modules/elastic-apm-http-client/index.js:1515:15)\n    at IncomingMessage.<anonymous> (/node_modules/elastic-apm-http-client/index.js:1049:17)\n    at IncomingMessage.emit (node:events:526:35)\n    at endReadableNT (node:internal/streams/readable:1376:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"},"message":"conclude intake request: error"} 
[myapp-0 myapp] {"log.level":"error","@timestamp":"2023-08-23T06:13:21.947Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (401): Unexpected APM Server response\n{\n  \"error\": \"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'\"\n}\n"} 

Kibana version: 8.9.1

Elasticsearch version:8.9.1

APM Server version: 8.9.1

APM Agent language and version: Node.JS

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[simitt (Silvia Mitter)]

The error log

authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)

indicates that authentication is enabled in the apm-server, but not in the apm agent. In the node code snippet that you shared, no secretToken or apiKey is configured, so requests to the apm-server will not be accepted. The nodejs docs show how to configure a token or api key in the elastic apm agent.

Also see how to create an ApiKey via Kibana UI. And finally, for APM Server on ECK specific commands, please check Connecting to APM Server on ECK.

[Kok_How_Teh]

@simitt

secretToken

If this is something required, I would like to use TLS instead. How to set that up?

[simitt (Silvia Mitter)]

Please refer to the ECK TLS docs for how to configure TLS, and to the docs on setting up TLS between apm agents and server.

[Kok_How_Teh]

Step 2: Configure the APM Server

I am using fleet-managed. Where the fuck is the "the path to the server certificate and key."!?! Can your documentation be improved to:

(1) Easily search for the information required
(2) Easy to navigate and follow instead of fragmented and piecewise information and the reader has to thread and piece all these together?

[Kok_How_Teh]

Where do I generate the TLS self-signed certs and where do I put them?

There are many resources created by the ECK and APM:

$ k get agent
NAME            HEALTH   AVAILABLE   EXPECTED   VERSION   AGE
elastic-agent   green    3           3          8.9.1     4d22h
fleet-server    green    3           3          8.9.1     4d22h

$ k get deployment
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
fleet-server-agent     3/3     3            3           4d22h
my-apm-server   3/3     3            3           4d1h

And there are elastic-agent-agent, fleet-server-agent, my-apm-server pods running. Here are my manifest:
apmserver.yml:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: my
  labels:
    app: apmserver
    component: eck
spec:
  version: 8.9.1
  count: 3
  secureSettings:
    - secretName: my-es-elastic-user
  config:
    output:
      elasticsearch:
        hosts: ["https://my-es-internal-http.default:9200"]
        username: elastic
        password: "${elastic}"
        protocol: "https"
        ssl.certificate_authorities:
          ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
  podTemplate:
    spec:
      containers:
        - name: apm-server
          volumeMounts:
            - mountPath: /usr/share/apm-server/config/elasticsearch-ca
              name: elasticsearch-ca
              readOnly: true
      volumes:
        - name: elasticsearch-ca
          secret:
            defaultMode: 420
            optional: false
            secretName: elasticsearch-eck-ca # This is the secret that holds the Elasticsearch CA cert

agents manifest:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: default
spec:
  version: 8.9.1
  kibanaRef:
    name: my
  elasticsearchRefs:
    - name: my
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 3
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: default
spec:
  version: 8.9.1
  kibanaRef:
    name: my
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0

Where should I mount the generated self-signed certs as k8s secret!?!

[Kok_How_Teh]

And where / how do we get the ca cert to configure the APM agent for step " Step 3: Configure APM agents"?

[axw (Andrew Wilkins)]

Please refrain from swearing at my colleague, and in these forums in general. I understand your frustration, and the documentation certainly needs to be improved. We'll take that on board.

Here's a script which deploys ECK, Elasticsearch, Kibana, and APM Server, and extracts the APM secret token and CA certificate.

#!/usr/bin/bash                                                                                                                   
set -e                                                                                                                            
                                                                                                                                  
# Install the ECK CRDs                                                                                                            
kubectl create -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml                                                      
                                                                                                                                  
# Install the ECK Operator                                                                                                        
kubectl apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml                                                    
                                                                                                                                  
# Create an Elasticsearch cluster                                                                                                 
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: elasticsearch.k8s.elastic.co/v1                                                                                       
kind: Elasticsearch                                                                                                               
metadata:                                                                                                                         
  name: quickstart                                                                                                                
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  nodeSets:                                                                                                                       
  - name: default                                                                                                                 
    count: 1                                                                                                                      
    config:                                                                                                                       
      node.store.allow_mmap: false                                                                                                
EOF                                                                                                                               
                                                                                                                                  
# Create a Kibana instance                                                                                                        
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: kibana.k8s.elastic.co/v1                                                                                              
kind: Kibana                                                                                                                      
metadata:                                                                                                                         
  name: quickstart                                                                                                                
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  count: 1                                                                                                                        
  elasticsearchRef:                                                                                                               
    name: quickstart                                                                                                              
EOF                                                                                                                               
                                                                                                                                  
# Create an APM Server.                                                                                                           
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: apm.k8s.elastic.co/v1                                                                                                 
kind: ApmServer                                                                                                                   
metadata:                                                                                                                         
  name: quickstart                                                                                                                
  namespace: default                                                                                                              
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  count: 1                                                                                                                        
  elasticsearchRef:                                                                                                               
    name: quickstart                                                                                                              
  kibanaRef:                                                                                                                      
    name: quickstart                                                                                                              
EOF                                                                                                                               
                                                                                                                                  
# The APM secret token is stored in a secret named "<name>-apm-token".                                                            
export ELASTIC_APM_SECRET_TOKEN=$(kubectl get secret/quickstart-apm-token --template '{{index .data "secret-token"}}' | base64 -d)
                                                                                                                                  
# The CA for APM Server's TLS certificate is stored in the secret "<name>-apm-http-certs-public".                                 
export ELASTIC_APM_SERVER_CA_CERT_FILE=$(mktemp)                                                                                  
kubectl get secret/quickstart-apm-http-certs-public \                                                                             
        --template '{{index .data "tls.crt"}}' | base64 -d > $ELASTIC_APM_SERVER_CA_CERT_FILE                                     
                                                                                                                                  
echo                                                                                                                              
echo "Elastic APM configuration:"                                                                                                 
echo                                                                                                                              
env | grep ELASTIC_APM_                                                                                                           

Running that you should end up with some output like:

...
Elastic APM configuration:

ELASTIC_APM_SECRET_TOKEN=p7x1C6aV6MtrC1I77Ly53G7w
ELASTIC_APM_SERVER_CA_CERT_FILE=/tmp/tmp.qRzQI9X0q8

To test, you can connect to APM Server from your host by port-forwarding to apm-server and using curl. e.g.

$ kubectl port-forward service/quickstart-apm-http 8200&
$ curl --resolve *:8200:127.0.0.1 --cacert /tmp/tmp.qRzQI9X0q8 -H "Authorization: Bearer p7x1C6aV6MtrC1I77Ly53G7w" https://quickstart-apm-http.default.apm.local:8200

Where should I mount the generated self-signed certs as k8s secret!?!

You'll need to mount the secrets in the pods running your instrumented applications, i.e. in the pod spec for your Node.js application. How you configure your application pods is naturally outside the scope of ECK -- you can use standard Kubernetes features to do this, referencing the secrets generated by ECK. See the script above for the secret names.

[Kok_How_Teh]

Where is the TLS key file?

[axw (Andrew Wilkins)]

What TLS key file? What do you need one for? ECK configures APM Server's TLS certificate and key. Agents do not need, and should not have access to, the key file. They just need the CA for verifying the server's TLS certificate, as described above.

[Kok_How_Teh]

Wasn't this shared by your colleague previously?

APM agent TLS communication | APM User Guide [8.9] | Elastic

SSL/TLS input settings | APM User Guide [8.9] | Elastic

[Kok_How_Teh]

What do you mean "What TLS key file"? Here is the required TLS cert and key files at the APM server (as shared by your colleague APM agent TLS communication | APM User Guide [8.9] | Elastic)

These certs are created in the k8s cluster:

$ k get secret | grep ca
elastic-agent-agent-fleetserver-ca                  Opaque                                2      8d
elasticsearch-eck-apm-ca                            Opaque                                1      26h
elasticsearch-eck-ca                                Opaque                                1      7d3h
fleet-server-agent-es-default-test-ca          Opaque                                2      8d
fleet-server-agent-http-ca-internal                 Opaque                                2      8d
test-apm-http-ca-internal                      Opaque                                2      7h8m
test-es-http-ca-internal                       Opaque                                2      8d
test-es-remote-ca                              Opaque                                1      8d
test-es-transport-ca-internal                  Opaque                                2      8d
test-kb-es-ca                                  Opaque                                2      22d
$ k get secret | grep cert
fleet-server-agent-http-certs-internal              Opaque                                3      8d
fleet-server-agent-http-certs-public                Opaque                                2      117d
test-apm-http-certs-internal                   Opaque                                3      7h9m
test-apm-http-certs-public                     Opaque                                2      7h9m
test-es-data-es-transport-certs                Opaque                                11     8d
test-es-http-certs-internal                    Opaque                                3      8d
test-es-http-certs-public                      Opaque                                2      22d
test-es-master-es-transport-certs              Opaque                                7      8d
test-es-transport-certs-public                 Opaque                                1      22d

What's the use case of each? Any documentation for that!?!

Mounted the public apm http cert in my app but to no avail!

[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-30T08:47:17.418Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"74fada477c4640a6d9a1af01828b4d04","timeline":[[5.335508,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":33142.227654409384,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n    at TLSSocket.emit (node:events:514:28)\n    at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n    at ssl.onhandshakedone (node:_tls_wrap:824:12)\n    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)"},"message":"conclude intake request: error"} 

[axw (Andrew Wilkins)]

What do you mean "What TLS key file"? Here is the required TLS cert and key files at the APM server (as shared by your colleague APM agent TLS communication | APM User Guide [8.9] | Elastic)

The screenshot you shared is of the APM Integration policy editor. As I mentioned in the other topic, the APM Integration is not relevant when you're using the ApmServer ECK object kind. On the APM agent TLS communication | APM User Guide [8.11] | Elastic page, the bits about SSL/TLS input settings are only relevant when using the Elastic Agent & Fleet Server to manage APM. So just ignore that, ECK has configured your APM Server with TLS.

What's the use case of each? Any documentation for that!?!

See TLS certificates | Elastic Cloud on Kubernetes [2.10] | Elastic

"The public certificate is stored in a secret named -[es|kb|apm|ent|agent]-http-certs-public."

See also the script I shared above.

Mounted the public apm http cert in my app but to no avail!

Can you please share the deployment/pod spec for your application? I will also try to put together a more complete example tomorrow, including an application instrumented with an APM agent.

[Kok_How_Teh]

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  serviceName: svc-myapp
  podManagementPolicy: OrderedReady
  replicas: 2
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: myapp # Has to match .spec.template.metadata.labels
      component: marketplace
  template:
    metadata:
      labels:
        app: myapp # Has to match .spec.selector.matchLabels
        component: marketplace
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - myapp
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: myapp
          image: <account_id>.dkr.ecr.<region>.amazonaws.com/myapp:latest
          imagePullPolicy: Always
          env:
            - name: NODE_ENV
              value: "production"
            - name: NODE_OPTIONS
              value: --max-old-space-size=16384
          ports:
            - containerPort: 443
              name: https
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health/ready
              port: https
              scheme: HTTPS
            initialDelaySeconds: 60
            periodSeconds: 5
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/live
              port: https
              scheme: HTTPS
            initialDelaySeconds: 60
            periodSeconds: 3
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          volumeMounts:
            - name: apm-public-cert
              mountPath: /etc/ca-certificates/tls.crt
              subPath: tls.crt
              readOnly: true
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
        - name: fluentd
          image: khteh/fluentd:latest
          env:
            - name: FLUENTD_ARGS
              value: -c /etc/td-agent/td-agent.conf
          volumeMounts:
            - name: log
              mountPath: /var/log/mylog
            - name: buyer-fluentd-config
              mountPath: /etc/td-agent
            - name: access-log-template
              mountPath: /tmp/access_log_template.json
              subPath: access_log_template.json
              readOnly: true
            - mountPath: /fluentd/elastic/tls.crt
              name: elasticsearch-ca
              subPath: tls.crt
              readOnly: true
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
      volumes:
        - name: buyer-fluentd-config
          configMap:
            name: buyer-fluentd-config
        - name: access-log-template
          configMap:
            name: access-log-template
        - name: log
          emptyDir: {}
        - name: elasticsearch-ca
          secret:
            defaultMode: 420
            optional: false
            secretName: elasticsearch-eck-ca # This is the secret that holds the Elasticsearch CA cert
        - name: apm-public-cert
          secret:
            defaultMode: 420
            optional: false
            secretName: apm-public-cert # This is the secret that holds the Elasticsearch CA cert

[Kok_How_Teh]

Doesn't work:

[buyerportal-0 buyerportal] {"log.level":"error","@timestamp":"2023-08-31T02:15:11.860Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (SELF_SIGNED_CERT_IN_CHAIN): self-signed certificate in certificate chain"} 

[buyerportal-0 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T02:14:34.279Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"e5a15d9440ce5025b8dd8a2a7be17ba8","timeline":[[4.879722,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":37575.4289468195,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n    at TLSSocket.emit (node:events:514:28)\n    at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n    at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"conclude intake request: error"} 

[axw (Andrew Wilkins)]

Can you try adding the ELASTIC_APM_SERVER_CA_CERT_FILE env var? (see Configuration options | APM Node.js Agent Reference [3.x] | Elastic)

I'm not a Node.js expert, but I think that the system CA certificates are not used by default. The agent docs seem to suggest that bundled Mozilla CA certs are used by default.

[Kok_How_Teh]

Isn't it enough to specify these at the elastic-node-apm configuration?

var apm = require('elastic-apm-node').start({
  // Override service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: 'my-apm',
  secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
  // Set custom APM Server URL (default: http://127.0.0.1:8200)
  serverUrl: 'https://my-apm-http:8200',
  // Only activate the agent if it's running in production
  active: process.env.NODE_ENV === 'production',
  //verifyServerCert: false,
  serverCaCertFile: '/etc/ca-certificates/tls.crt',
  logLevel: 'trace'
})

[Kok_How_Teh]

Just did. Doesn't help at all.

[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T06:41:18.968Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"13284993d0653d13ec672d612061cd46","concluded":false,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n    at TLSSocket.emit (node:events:514:28)\n    at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n    at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"completePart intakeReq"} 
[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T06:41:18.968Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"13284993d0653d13ec672d612061cd46","timeline":[[4.905638,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":39428.85776797586,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n    at TLSSocket.emit (node:events:514:28)\n    at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n    at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"conclude intake request: error"} 

[axw (Andrew Wilkins)]

@Kok_How_Teh ah yes, I missed that in your initial config. Using serverCaCertFile is fine, you don't need to add the env var then.

This works for me:

apiVersion: v1
kind: Pod
metadata:
  name: tlsdemo
spec:
  volumes:
  - name: apm-certs
    secret:
      defaultMode: 420
      optional: false
      secretName: quickstart-apm-http-certs-public
  containers:
  - name: app
    image: app
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 3000
    volumeMounts:
    - name: apm-certs
      mountPath: /etc/ca-certificates/tls.crt
      subPath: tls.crt
      readOnly: true
    env:
    - name: ELASTIC_APM_SERVER_URL
      value: https://quickstart-apm-http:8200
    - name: ELASTIC_APM_SECRET_TOKEN
      valueFrom:
        secretKeyRef:
          name: quickstart-apm-token
          key: secret-token

Code:

const apm = require('elastic-apm-node').start({
  serviceName: 'tls_demo',
  serverCaCertFile: '/etc/ca-certificates/tls.crt'
})

const express = require('express')
const app = express()
const port = 3000

app.get('/', function (req, res) {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Maybe the "apm-public-cert" secret does not hold the correct certificates?

[Kok_How_Teh]

They do match.