Kibana version:
7.13.4
Elasticsearch version:
7.13.4
APM Server version:
7.13.4
APM Agent language and version:
NA
Browser version:
NA
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Helm
Fresh install or upgraded from other version?
Fresh
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
NA
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We set api_key.enabled: true in API server then creates API key in APM server for the Open Telemetry agent but they are unable to connect to APM Server over secure TLS connection. APM server is running fine and is connected to Elasticsearch over TLS connection.
Steps to reproduce:
- Install Otel agent
Errors in browser console (if relevant):
NA
Provide logs and/or server output (if relevant):
Below is the yaml configuration of Otel Agent Daemonset
---
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-agent-conf
namespace: es
labels:
app: opentelemetry
component: otel-agent-conf
data:
otel-agent-config: |
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
load:
memory:
otlp:
protocols:
grpc:
http:
jaeger:
protocols:
grpc:
thrift_compact:
thrift_http:
zipkin:
exporters:
otlp/elastic:
endpoint: "https://apm-server.es.svc.cluster.local:8200"
headers:
# Elastic APM Server API key
Authorization: "ApiKey ${ELASTIC_APM_SERVER_APIKEY}"
logging:
loglevel: WARN
processors:
batch:
memory_limiter:
# Same as --mem-ballast-size-mib CLI argument
ballast_size_mib: 165
# 80% of maximum memory up to 2G
limit_mib: 400
# 25% of limit up to 2G
spike_limit_mib: 100
check_interval: 5s
service:
pipelines:
metrics:
receivers: [otlp, hostmetrics]
processors: [batch]
exporters: [otlp/elastic, logging]
traces:
receivers: [otlp, jaeger, zipkin]
processors: [memory_limiter, batch]
exporters: [otlp/elastic, logging]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: otel-agent
namespace: es
labels:
app: opentelemetry
component: otel-agent
spec:
selector:
matchLabels:
app: opentelemetry
component: otel-agent
template:
metadata:
labels:
app: opentelemetry
component: otel-agent
spec:
containers:
- command:
- "/otelcol"
- "--config=/conf/otel-agent-config.yaml"
# Memory Ballast size should be max 1/3 to 1/2 of memory.
# - "--mem-ballast-size-mib=165"
image: 470776511283.dkr.ecr.ap-south-1.amazonaws.com/dev-reco-otel:latest
name: otel-agent
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6831 # Jaeger Thrift Compact
protocol: UDP
- containerPort: 8888 # Prometheus Metrics
- containerPort: 9411 # Default endpoint for Zipkin receiver.
- containerPort: 14250 # Default endpoint for Jaeger gRPC receiver.
- containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
- containerPort: 4317 # Default OpenTelemetry gRPC receiver port.
- containerPort: 55681 # Default OpenTelemetry HTTP receiver port.
env:
# Get pod ip so that k8s_tagger can tag resources
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
# This is picked up by the resource detector
- name: OTEL_RESOURCE_ATTRIBUTES
value: "k8s.pod.ip=$(POD_IP)"
- name: ELASTIC_APM_SERVER_APIKEY
valueFrom:
secretKeyRef:
name: elastic-apm-server-key
key: encryptionkey
volumeMounts:
- name: otel-agent-config-vol
mountPath: /conf
livenessProbe:
httpGet:
path: /
port: 13133 # Health Check extension default port.
readinessProbe:
httpGet:
path: /
port: 13133 # Health Check extension default port.
volumes:
- configMap:
name: otel-agent-conf
items:
- key: otel-agent-config
path: otel-agent-config.yaml
name: otel-agent-config-vol
Below is the error in one of the otel agent pod
2021-08-01T12:08:26.594Z info service/collector.go:211 Everything is ready. Begin running and processing data.
2021-08-01T12:08:36.610Z info exporterhelper/queued_retry.go:325 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "otlp/elastic", "error": "failed to push metrics data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"", "interval": "5.52330144s"}
2021-08-01T12:08:46.632Z info exporterhelper/queued_retry.go:325 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "otlp/elastic", "error": "failed to push metrics data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"", "interval": "5.822800266s"}
2021-08-01T12:08:48.416Z info service/collector.go:225 Received signal from OS {"signal": "terminated"}
2021-08-01T12:08:48.416Z info service/collector.go:331 Starting shutdown...
Please help us to achieve the connection.
