# How to get ECK APM to work with any type of agent at all!?!

**URL:** <https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429>\
**Category:** APM\
**Tags:** docker, nodejs, server\
**Created:** [August 23, 2023, 6:14am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429 "2023-08-23T06:14:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 23, 2023, 6:14am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/1 "2023-08-23T06:14:59Z")

</div>

```auto
var apm = require('elastic-apm-node').start({
  serviceName: 'my-apm',
  secretToken: '',
  // Set custom APM Server URL (default: http://127.0.0.1:8200)
  serverUrl: 'https://my-apm-http:8200',
  // Only activate the agent if it's running in production
  active: process.env.NODE_ENV === 'production',
  verifyServerCert: false,
  serverCaCertFile: '/etc/ca-certificates/apm.crt',

```

Errors:

```auto
[myapp-0 myapp] {"log.level":"trace","@timestamp":"2023-08-23T06:13:21.947Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"83fb5052fa3f5a4420824fabac133671","timeline":[[10002.293778,"completePart gzipStream",null],[10002.594401,"completePart intakeReq",null],[10003.133811,"completePart intakeRes","Unexpected APM Server response"]],"bytesWritten":731,"backoffDelayMs":15952.873427495506,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"Unexpected APM Server response","stack_trace":"Error: Unexpected APM Server response\n at processIntakeErrorResponse (/node_modules/elastic-apm-http-client/index.js:1515:15)\n at IncomingMessage.<anonymous> (/node_modules/elastic-apm-http-client/index.js:1049:17)\n at IncomingMessage.emit (node:events:526:35)\n at endReadableNT (node:internal/streams/readable:1376:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"},"message":"conclude intake request: error"} 
[myapp-0 myapp] {"log.level":"error","@timestamp":"2023-08-23T06:13:21.947Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (401): Unexpected APM Server response\n{\n \"error\": \"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'\"\n}\n"} 

```

**Kibana version** : 8.9.1

**Elasticsearch version** :8.9.1

**APM Server version** : 8.9.1

**APM Agent language and version** : Node.JS

**Browser version** :

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:

**Fresh install or upgraded from other version?** Fresh

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

**Steps to reproduce** :  
1.  
2.  
3.

**Errors in browser console (if relevant)**:

**Provide logs and/or server output (if relevant)**:

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [August 23, 2023, 8:02am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/2 "2023-08-23T08:02:46Z")

</div>

The error log

```auto
authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)

```

indicates that authentication is enabled in the apm-server, but not in the apm agent. In the `node` code snippet that you shared, no `secretToken` or `apiKey` is configured, so requests to the apm-server will not be accepted. The [nodejs docs](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuring-the-agent.html#agent-configuration-object) show how to configure a token or api key in the elastic apm agent.

Also see [how to create an ApiKey via Kibana UI](https://www.elastic.co/guide/en/apm/guide/current/api-key.html#create-an-api-key). And finally, for APM Server on ECK specific commands, please check [Connecting to APM Server on ECK](https://www.elastic.co/guide/en/cloud-on-k8s/2.8/k8s-apm-connecting.html#k8s-apm-service).

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 23, 2023, 8:20am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/3 "2023-08-23T08:20:48Z")

</div>

> [@simitt](#):
>
> `secretToken`

If this is something required, I would like to use TLS instead. How to set that up?

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [August 23, 2023, 11:25am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/4 "2023-08-23T11:25:35Z")

</div>

Please refer to the [ECK TLS](https://www.elastic.co/guide/en/cloud-on-k8s/2.8/k8s-apm-advanced-configuration.html#k8s-apm-tls) docs for how to configure TLS, and to the docs on [setting up TLS between apm agents and server](https://www.elastic.co/guide/en/apm/guide/current/agent-tls.html).

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 26, 2023, 7:15am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/5 "2023-08-26T07:15:29Z")

</div>

##### Step 2: Configure the APM Server

I am using fleet-managed. Where the fuck is the "the path to the server certificate and key."!?! Can your documentation be improved to:

(1) Easily search for the information required  
(2) Easy to navigate and follow instead of fragmented and piecewise information and the reader has to thread and piece all these together?

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 27, 2023, 6:57am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/6 "2023-08-27T06:57:38Z")

</div>

Where do I generate the TLS self-signed certs and where do I put them?

There are many resources created by the ECK and APM:

```auto
$ k get agent
NAME HEALTH AVAILABLE EXPECTED VERSION AGE
elastic-agent green 3 3 8.9.1 4d22h
fleet-server green 3 3 8.9.1 4d22h

```

```auto
$ k get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
fleet-server-agent 3/3 3 3 4d22h
my-apm-server 3/3 3 3 4d1h

```

And there are `elastic-agent-agent`, `fleet-server-agent`, `my-apm-server` pods running. Here are my manifest:  
`apmserver.yml`:

```auto
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: my
  labels:
    app: apmserver
    component: eck
spec:
  version: 8.9.1
  count: 3
  secureSettings:
    - secretName: my-es-elastic-user
  config:
    output:
      elasticsearch:
        hosts: ["https://my-es-internal-http.default:9200"]
        username: elastic
        password: "${elastic}"
        protocol: "https"
        ssl.certificate_authorities:
          ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
  podTemplate:
    spec:
      containers:
        - name: apm-server
          volumeMounts:
            - mountPath: /usr/share/apm-server/config/elasticsearch-ca
              name: elasticsearch-ca
              readOnly: true
      volumes:
        - name: elasticsearch-ca
          secret:
            defaultMode: 420
            optional: false
            secretName: elasticsearch-eck-ca # This is the secret that holds the Elasticsearch CA cert

```

agents manifest:

```auto
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: default
spec:
  version: 8.9.1
  kibanaRef:
    name: my
  elasticsearchRefs:
    - name: my
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 3
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: default
spec:
  version: 8.9.1
  kibanaRef:
    name: my
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0

```

Where should I mount the generated self-signed certs as k8s secret!?!

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 28, 2023, 3:33am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/7 "2023-08-28T03:33:25Z")

</div>

And where / how do we get the ca cert to configure the APM agent for step " Step 3: Configure APM agents"?

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 29, 2023, 2:34am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/8 "2023-08-29T02:34:29Z")

</div>

Please refrain from swearing at my colleague, and in these forums in general. I understand your frustration, and the documentation certainly needs to be improved. We'll take that on board.

Here's a script which deploys ECK, Elasticsearch, Kibana, and APM Server, and extracts the APM secret token and CA certificate.

```auto
#!/usr/bin/bash                                                                                                                   
set -e                                                                                                                            
                                                                                                                                  
# Install the ECK CRDs                                                                                                            
kubectl create -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml                                                      
                                                                                                                                  
# Install the ECK Operator                                                                                                        
kubectl apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml                                                    
                                                                                                                                  
# Create an Elasticsearch cluster                                                                                                 
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: elasticsearch.k8s.elastic.co/v1                                                                                       
kind: Elasticsearch                                                                                                               
metadata:                                                                                                                         
  name: quickstart                                                                                                                
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  nodeSets:                                                                                                                       
  - name: default                                                                                                                 
    count: 1                                                                                                                      
    config:                                                                                                                       
      node.store.allow_mmap: false                                                                                                
EOF                                                                                                                               
                                                                                                                                  
# Create a Kibana instance                                                                                                        
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: kibana.k8s.elastic.co/v1                                                                                              
kind: Kibana                                                                                                                      
metadata:                                                                                                                         
  name: quickstart                                                                                                                
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  count: 1                                                                                                                        
  elasticsearchRef:                                                                                                               
    name: quickstart                                                                                                              
EOF                                                                                                                               
                                                                                                                                  
# Create an APM Server.                                                                                                           
cat <<EOF | kubectl apply -f -                                                                                                    
apiVersion: apm.k8s.elastic.co/v1                                                                                                 
kind: ApmServer                                                                                                                   
metadata:                                                                                                                         
  name: quickstart                                                                                                                
  namespace: default                                                                                                              
spec:                                                                                                                             
  version: 8.9.1                                                                                                                  
  count: 1                                                                                                                        
  elasticsearchRef:                                                                                                               
    name: quickstart                                                                                                              
  kibanaRef:                                                                                                                      
    name: quickstart                                                                                                              
EOF                                                                                                                               
                                                                                                                                  
# The APM secret token is stored in a secret named "<name>-apm-token".                                                            
export ELASTIC_APM_SECRET_TOKEN=$(kubectl get secret/quickstart-apm-token --template '{{index .data "secret-token"}}' | base64 -d)
                                                                                                                                  
# The CA for APM Server's TLS certificate is stored in the secret "<name>-apm-http-certs-public".                                 
export ELASTIC_APM_SERVER_CA_CERT_FILE=$(mktemp)                                                                                  
kubectl get secret/quickstart-apm-http-certs-public \                                                                             
        --template '{{index .data "tls.crt"}}' | base64 -d > $ELASTIC_APM_SERVER_CA_CERT_FILE                                     
                                                                                                                                  
echo                                                                                                                              
echo "Elastic APM configuration:"                                                                                                 
echo                                                                                                                              
env | grep ELASTIC_APM_                                                                                                           

```

Running that you should end up with some output like:

```bash
...
Elastic APM configuration:

ELASTIC_APM_SECRET_TOKEN=p7x1C6aV6MtrC1I77Ly53G7w
ELASTIC_APM_SERVER_CA_CERT_FILE=/tmp/tmp.qRzQI9X0q8

```

To test, you can connect to APM Server from your host by port-forwarding to apm-server and using `curl`. e.g.

```auto
$ kubectl port-forward service/quickstart-apm-http 8200&
$ curl --resolve *:8200:127.0.0.1 --cacert /tmp/tmp.qRzQI9X0q8 -H "Authorization: Bearer p7x1C6aV6MtrC1I77Ly53G7w" https://quickstart-apm-http.default.apm.local:8200

```

> Where should I mount the generated self-signed certs as k8s secret!?!

You'll need to mount the secrets in the pods running your instrumented applications, i.e. in the pod spec for your Node.js application. How you configure your application pods is naturally outside the scope of ECK -- you can use standard Kubernetes features to do this, referencing the secrets generated by ECK. See the script above for the secret names.

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 29, 2023, 6:31am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/9 "2023-08-29T06:31:37Z")

</div>

Where is the TLS key file?

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 29, 2023, 6:46am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/10 "2023-08-29T06:46:42Z")

</div>

What TLS key file? What do you need one for? ECK configures APM Server's TLS certificate and key. Agents do not need, and should not have access to, the key file. They just need the CA for verifying the server's TLS certificate, as described above.

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 29, 2023, 6:51am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/11 "2023-08-29T06:51:16Z")

</div>

Wasn't this shared by your colleague previously?

> **[APM agent TLS communication | APM User Guide \[8.9\] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/agent-tls.html#agent-self-sign-3)**

> **[SSL/TLS input settings | APM User Guide \[8.9\] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/agent-server-ssl.html)**

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 30, 2023, 8:49am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/13 "2023-08-30T08:49:24Z")

</div>

What do you mean "What TLS key file"? Here is the required TLS cert and key files at the APM server (as shared by your colleague [APM agent TLS communication | APM User Guide [8.9] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/agent-tls.html))

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/5/556c5a7e7d97981c4c2ed79ffa6153990b30b2e3.png)

These certs are created in the k8s cluster:

```auto
$ k get secret | grep ca
elastic-agent-agent-fleetserver-ca Opaque 2 8d
elasticsearch-eck-apm-ca Opaque 1 26h
elasticsearch-eck-ca Opaque 1 7d3h
fleet-server-agent-es-default-test-ca Opaque 2 8d
fleet-server-agent-http-ca-internal Opaque 2 8d
test-apm-http-ca-internal Opaque 2 7h8m
test-es-http-ca-internal Opaque 2 8d
test-es-remote-ca Opaque 1 8d
test-es-transport-ca-internal Opaque 2 8d
test-kb-es-ca Opaque 2 22d
$ k get secret | grep cert
fleet-server-agent-http-certs-internal Opaque 3 8d
fleet-server-agent-http-certs-public Opaque 2 117d
test-apm-http-certs-internal Opaque 3 7h9m
test-apm-http-certs-public Opaque 2 7h9m
test-es-data-es-transport-certs Opaque 11 8d
test-es-http-certs-internal Opaque 3 8d
test-es-http-certs-public Opaque 2 22d
test-es-master-es-transport-certs Opaque 7 8d
test-es-transport-certs-public Opaque 1 22d

```

What's the use case of each? Any documentation for that!?!

Mounted the public apm http cert in my app but to no avail!

```auto
[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-30T08:47:17.418Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"74fada477c4640a6d9a1af01828b4d04","timeline":[[5.335508,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":33142.227654409384,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n at TLSSocket.emit (node:events:514:28)\n at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n at ssl.onhandshakedone (node:_tls_wrap:824:12)\n at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)"},"message":"conclude intake request: error"} 

```

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 30, 2023, 10:02am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/14 "2023-08-30T10:02:39Z")

</div>

> What do you mean "What TLS key file"? Here is the required TLS cert and key files at the APM server (as shared by your colleague [APM agent TLS communication | APM User Guide [8.9] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/agent-tls.html))

The screenshot you shared is of the APM Integration policy editor. As I mentioned in the [other topic](https://discuss.elastic.co/t/authentication-failed-missing-or-improperly-formatted-authorization-header-expected-authorization-bearer-secret-token-or-authorization-apikey-base64-api-key-id-api-key/340321/7), the APM Integration is not relevant when you're using the `ApmServer` ECK object kind. On the [APM agent TLS communication | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/agent-tls.html) page, the bits about `SSL/TLS input settings` are only relevant when using the Elastic Agent & Fleet Server to manage APM. So just ignore that, ECK has configured your APM Server with TLS.

> What's the use case of each? Any documentation for that!?!

See [TLS certificates | Elastic Cloud on Kubernetes [2.10] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html)

"The public certificate is stored in a secret named -[es|kb|apm|ent|agent]-http-certs-public."

See also the script I shared above.

> Mounted the public apm http cert in my app but to no avail!

Can you please share the deployment/pod spec for your application? I will also try to put together a more complete example tomorrow, including an application instrumented with an APM agent.

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 30, 2023, 10:22am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/15 "2023-08-30T10:22:26Z")

</div>

```auto
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  serviceName: svc-myapp
  podManagementPolicy: OrderedReady
  replicas: 2
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: myapp # Has to match .spec.template.metadata.labels
      component: marketplace
  template:
    metadata:
      labels:
        app: myapp # Has to match .spec.selector.matchLabels
        component: marketplace
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - myapp
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: myapp
          image: <account_id>.dkr.ecr.<region>.amazonaws.com/myapp:latest
          imagePullPolicy: Always
          env:
            - name: NODE_ENV
              value: "production"
            - name: NODE_OPTIONS
              value: --max-old-space-size=16384
          ports:
            - containerPort: 443
              name: https
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health/ready
              port: https
              scheme: HTTPS
            initialDelaySeconds: 60
            periodSeconds: 5
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/live
              port: https
              scheme: HTTPS
            initialDelaySeconds: 60
            periodSeconds: 3
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          volumeMounts:
            - name: apm-public-cert
              mountPath: /etc/ca-certificates/tls.crt
              subPath: tls.crt
              readOnly: true
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
        - name: fluentd
          image: khteh/fluentd:latest
          env:
            - name: FLUENTD_ARGS
              value: -c /etc/td-agent/td-agent.conf
          volumeMounts:
            - name: log
              mountPath: /var/log/mylog
            - name: buyer-fluentd-config
              mountPath: /etc/td-agent
            - name: access-log-template
              mountPath: /tmp/access_log_template.json
              subPath: access_log_template.json
              readOnly: true
            - mountPath: /fluentd/elastic/tls.crt
              name: elasticsearch-ca
              subPath: tls.crt
              readOnly: true
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
      volumes:
        - name: buyer-fluentd-config
          configMap:
            name: buyer-fluentd-config
        - name: access-log-template
          configMap:
            name: access-log-template
        - name: log
          emptyDir: {}
        - name: elasticsearch-ca
          secret:
            defaultMode: 420
            optional: false
            secretName: elasticsearch-eck-ca # This is the secret that holds the Elasticsearch CA cert
        - name: apm-public-cert
          secret:
            defaultMode: 420
            optional: false
            secretName: apm-public-cert # This is the secret that holds the Elasticsearch CA cert

```

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 31, 2023, 2:15am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/16 "2023-08-31T02:15:42Z")

</div>

Doesn't work:

```auto
[buyerportal-0 buyerportal] {"log.level":"error","@timestamp":"2023-08-31T02:15:11.860Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (SELF_SIGNED_CERT_IN_CHAIN): self-signed certificate in certificate chain"} 

[buyerportal-0 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T02:14:34.279Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"e5a15d9440ce5025b8dd8a2a7be17ba8","timeline":[[4.879722,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":37575.4289468195,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n at TLSSocket.emit (node:events:514:28)\n at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"conclude intake request: error"} 

```

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 31, 2023, 6:01am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/17 "2023-08-31T06:01:52Z")

</div>

Can you try adding the ELASTIC\_APM\_SERVER\_CA\_CERT\_FILE env var? (see [Configuration options | APM Node.js Agent Reference [3.x] | Elastic](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#server-ca-cert-file))

I'm not a Node.js expert, but I _think_ that the system CA certificates are not used by default. The agent docs seem to suggest that bundled Mozilla CA certs are used by default.

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 31, 2023, 6:32am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/18 "2023-08-31T06:32:03Z")

</div>

Isn't it enough to specify these at the elastic-node-apm configuration?

```auto
var apm = require('elastic-apm-node').start({
  // Override service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: 'my-apm',
  secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
  // Set custom APM Server URL (default: http://127.0.0.1:8200)
  serverUrl: 'https://my-apm-http:8200',
  // Only activate the agent if it's running in production
  active: process.env.NODE_ENV === 'production',
  //verifyServerCert: false,
  serverCaCertFile: '/etc/ca-certificates/tls.crt',
  logLevel: 'trace'
})

```

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 31, 2023, 6:42am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/19 "2023-08-31T06:42:12Z")

</div>

Just did. Doesn't help at all.

```auto
[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T06:41:18.968Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"13284993d0653d13ec672d612061cd46","concluded":false,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n at TLSSocket.emit (node:events:514:28)\n at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"completePart intakeReq"} 
[buyerportal-1 buyerportal] {"log.level":"trace","@timestamp":"2023-08-31T06:41:18.968Z","log":{"logger":"elastic-apm-node"},"event.module":"apmclient","reqId":"13284993d0653d13ec672d612061cd46","timeline":[[4.905638,"completePart intakeReq","self-signed certificate in certificate chain"]],"bytesWritten":10,"backoffDelayMs":39428.85776797586,"ecs":{"version":"1.6.0"},"error":{"type":"Error","message":"self-signed certificate in certificate chain","stack_trace":"Error: self-signed certificate in certificate chain\n at TLSSocket.onConnectSecure (node:_tls_wrap:1627:34)\n at TLSSocket.emit (node:events:514:28)\n at TLSSocket._finishInit (node:_tls_wrap:1038:8)\n at ssl.onhandshakedone (node:_tls_wrap:824:12)"},"message":"conclude intake request: error"} 

```

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 31, 2023, 6:46am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/20 "2023-08-31T06:46:32Z")

</div>

@khteh ah yes, I missed that in your initial config. Using `serverCaCertFile` is fine, you don't need to add the env var then.

This works for me:

```auto
apiVersion: v1
kind: Pod
metadata:
  name: tlsdemo
spec:
  volumes:
  - name: apm-certs
    secret:
      defaultMode: 420
      optional: false
      secretName: quickstart-apm-http-certs-public
  containers:
  - name: app
    image: app
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 3000
    volumeMounts:
    - name: apm-certs
      mountPath: /etc/ca-certificates/tls.crt
      subPath: tls.crt
      readOnly: true
    env:
    - name: ELASTIC_APM_SERVER_URL
      value: https://quickstart-apm-http:8200
    - name: ELASTIC_APM_SECRET_TOKEN
      valueFrom:
        secretKeyRef:
          name: quickstart-apm-token
          key: secret-token

```

Code:

```auto
const apm = require('elastic-apm-node').start({
  serviceName: 'tls_demo',
  serverCaCertFile: '/etc/ca-certificates/tls.crt'
})

const express = require('express')
const app = express()
const port = 3000

app.get('/', function (req, res) {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

```

Maybe the "apm-public-cert" secret does not hold the correct certificates?

---

<div class="post-metadata">

**Author:** ![khteh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khteh/32/147573_2.png) [@khteh](https://discuss.elastic.co/u/khteh)\
**Post date:** [August 31, 2023, 6:53am UTC](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429/21 "2023-08-31T06:53:42Z")

</div>

They do match.

[Next page](https://discuss.elastic.co/t/how-to-get-eck-apm-to-work-with-any-type-of-agent-at-all/341429.md?page=2)