Is it possible to use APM server without TLS to receive OpenTelemetry data?

Elastic Observability APM
1

Install OpenTelemetry Operator

helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"

Create OpenTelemetry Collector

cat <<EOF | kubectl apply -f -
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: ":4318"
          grpc:
            endpoint: ":4317"

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 2000

    exporters:
      otlp/elastic:
        endpoint: apm-server-quickstart-apm-http.default.svc.cluster.local:8200
        headers:
          Authorization: "Bearer elastic"

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [otlp/elastic]
EOF

Install APM server
(Elasticsearch and Kibana have been installed with the same version in ECK CRD)

cat <<EOF | kubectl apply -f -
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server-quickstart
  namespace: default
spec:
  version: 8.13.4
  count: 1
  elasticsearchRef:
    name: quickstart
  kibanaRef: 
    name: quickstart
EOF

Create APM Agent for OpenTelemetry in Kibana

agent
agent1521×1218 138 KB

Port forward

kubectl port-forward service/apm-server-quickstart-apm-http 8200
kubectl port-forward service/my-collector-collector 4317:4317
kubectl port-forward service/my-collector-collector 4318:4318

Method 1

Set OTEL environment variables and start an application
(Isn't there a variable for trace like OTEL_TRACES_EXPORTER?)

export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=service.name=testService,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer elastic"
node app.js

Access the application.
There isn't any log sent to APM server.

Method 2

Set OTEL environment variables and start an application

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
export OTEL_SERVICE_NAME="test-service"
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
node app.js

Access the application.
Got these logs in OpenTelemetry Collecotor's my-collector-collector pod:

2024-07-26T08:33:49.989Z	warn	zapgrpc/zapgrpc.go:193	[core] [Channel #1 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "10.43.99.11:8200", ServerName: "apm-server-quickstart-apm-http.default.svc.cluster.local:8200", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for apm-server-quickstart-apm-http.default.apm.local, apm-server-quickstart-apm-http, apm-server-quickstart-apm-http.default.svc, apm-server-quickstart-apm-http.default, not apm-server-quickstart-apm-http.default.svc.cluster.local"	{"grpc_log": true}
2024-07-26T08:33:49.989Z	info	exporterhelper/retry_sender.go:118	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for apm-server-quickstart-apm-http.default.apm.local, apm-server-quickstart-apm-http, apm-server-quickstart-apm-http.default.svc, apm-server-quickstart-apm-http.default, not apm-server-quickstart-apm-http.default.svc.cluster.local\"", "interval": "4.620562738s"}

If use this in OpenTelemetryCollector:

    exporters:
      otlp/elastic:
        endpoint: apm-server-quickstart-apm-http:8200
        headers:
          Authorization: "Bearer elastic"

Got

2024-07-26T08:48:52.383Z	warn	zapgrpc/zapgrpc.go:193	[core] [Channel #2 SubChannel #7]grpc: addrConn.createTransport failed to connect to {Addr: "10.43.99.11:8200", ServerName: "apm-server-quickstart-apm-http:8200", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"	{"grpc_log": true}
2024-07-26T08:48:52.384Z	info	exporterhelper/retry_sender.go:118	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "4.247949438s"}

If use this in OpenTelemetryCollector:

    exporters:
      otlp/elastic:
        endpoint: apm-server-quickstart-apm-http:8200
        headers:
          Authorization: "Bearer elastic"
        tls:
          insecure: true

Got

2024-07-26T08:52:32.436Z	warn	zapgrpc/zapgrpc.go:193	[core] [Channel #2 SubChannel #7]grpc: addrConn.createTransport failed to connect to {Addr: "10.43.99.11:8200", ServerName: "apm-server-quickstart-apm-http:8200", }. Err: connection error: desc = "error reading server preface: EOF"	{"grpc_log": true}
2024-07-26T08:52:32.436Z	info	exporterhelper/retry_sender.go:118	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: EOF\"", "interval": "5.575712041s"}

So if don't use TLS, is it possible to access APM server?
If necessary, how to do with cert-manager?

2

Hi,

I had same issue. You can use 'insecure_skip_verify: true' to resolve this problem.

OTEL Collector contrib example:

      otlp/elastic:
        # Elastic APM server https endpoint without the "https://" prefix
        endpoint: "apm-server-apm-http.elastic-system:8200"
        tls:
          insecure: false
          insecure_skip_verify: true
        headers:
            ......auth stuff.....

Also you need to add to Kibana manifest xpack fleet app package:
Documentation Link

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic-system
spec:
  version: 8.14.3
  count: 1
  config:
    xpack.fleet.packages:
      - name: apm
        version: latest
3

@Martin_Slanina
Thank you. I tried that, but in my environment I got this error in my-collector-collecotr pod:

{"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = authentication failed", "dropped_items": 16}
4

Did you take Kubernetes secret apm-server-apm-token -> secret-token and put it into Authorization header in exporter config?:

....
 endpoint: apm-server-quickstart-apm-http:8200
   headers:
     Authorization: "Bearer xxxxxxx" <------ secret token
...
5

@Martin_Slanina
Yes, as I wrote in the question, I set the secret to operator exporter directly:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: ":4318"
          grpc:
            endpoint: ":4317"

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 2000

    exporters:
      otlp/elastic:
        endpoint: apm-server-quickstart-apm-http.default.svc.cluster.local:8200
        headers:
          Authorization: "Bearer elastic"
        tls:
          insecure: false
          insecure_skip_verify: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [otlp/elastic]