# Can't connect OpenTelemetry collector with APM Server **URL:** \ **Category:** APM\ **Tags:** elastic-stack-security, docker, open-telemetry\ **Created:** [October 16, 2023, 5:04am UTC](https://discuss.elastic.co/t/cant-connect-opentelemetry-collector-with-apm-server/345064 "2023-10-16T05:04:18Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![sakibul.munna](https://avatars.discourse-cdn.com/v4/letter/s/8baadc/32.png) [@sakibul.munna](https://discuss.elastic.co/u/sakibul.munna)\ **Post date:** [October 16, 2023, 5:04am UTC](https://discuss.elastic.co/t/cant-connect-opentelemetry-collector-with-apm-server/345064/1 "2023-10-16T05:04:18Z")
**Kibana version** : 8.10.2 **Elasticsearch version** : 8.10.2 **APM Server version** : 8.10.2 **APM Agent language and version** : **Browser version** : Chrome Version 118.0.5993.70 (Official Build) (64-bit) **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: I am running elk stack and apm server using docker. **Fresh install or upgraded from other version?** Fresh **Is there anything special in your setup?** No **Description of the problem:** I am trying to connect my OpenTelemetry collector to the APM server. But getting this error from the collector service, ```auto collector | 2023-10-16T04:35:53.027Z warn zapgrpc/zapgrpc.go:195 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "apm-server:8200", ServerName: "apm-server:8200", }. Err: connection error: desc = "error reading server preface: remote error: tls: bad certificate" {"grpc_log": true} ``` - My docker-compose file to run the elk stack, apm server, and collector is below, ```auto version: "3.8" volumes: certs: driver: local esdata01: driver: local kibanadata: driver: local logstashdata01: driver: local apmdata: driver: local services: setup: image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} volumes: - certs:/usr/share/elasticsearch/config/certs user: "0" command: > bash -c ' if [x${ELASTIC_PASSWORD} == x]; then echo "Set the ELASTIC_PASSWORD environment variable in the .env file"; exit 1; elif [x${KIBANA_PASSWORD} == x]; then echo "Set the KIBANA_PASSWORD environment variable in the .env file"; exit 1; elif [x${APM_PASSWORD} == x]; then echo "Set the APM_PASSWORD environment variable in the .env file"; exit 1; fi; if [! -f config/certs/ca.zip]; then echo "Creating CA"; bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip; unzip config/certs/ca.zip -d config/certs; fi; if [! -f config/certs/certs.zip]; then echo "Creating certs"; echo -ne \ "instances:\n"\ " - name: es01\n"\ " dns:\n"\ " - es01\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ " - name: kibana\n"\ " dns:\n"\ " - kibana\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ " - name: apm-server\n"\ " dns:\n"\ " - apm-server\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ > config/certs/instances.yml; bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key; unzip config/certs/certs.zip -d config/certs; fi; echo "Setting file permissions" chown -R root:root config/certs; find . -type d -exec chmod 755 \{\} \;; find . -type f -exec chmod 644 \{\} \;; echo "Waiting for Elasticsearch availability"; until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done; echo "Setting kibana_system password"; until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done; echo "Setting apm_system password"; until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/apm_system/_password -d "{\"password\":\"${APM_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done; echo "All done!"; ' healthcheck: test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt] && [-f config/certs/kibana/kibana.crt] && [-f config/certs/apm-server/apm-server.crt]" ] interval: 1s timeout: 5s retries: 120 networks: - elk es01: depends_on: setup: condition: service_healthy image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} labels: co.elastic.logs/module: elasticsearch volumes: - certs:/usr/share/elasticsearch/config/certs - esdata01:/usr/share/elasticsearch/data ports: - ${ES_PORT}:9200 environment: - node.name=es01 - cluster.name=${CLUSTER_NAME} - discovery.type=single-node - ELASTIC_PASSWORD=${ELASTIC_PASSWORD} - bootstrap.memory_lock=true - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/es01/es01.key - xpack.security.http.ssl.certificate=certs/es01/es01.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.enabled=true - xpack.security.transport.ssl.key=certs/es01/es01.key - xpack.security.transport.ssl.certificate=certs/es01/es01.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} mem_limit: ${ES_MEM_LIMIT} ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ "CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'", ] interval: 10s timeout: 10s retries: 120 networks: - elk kibana: depends_on: es01: condition: service_healthy image: docker.elastic.co/kibana/kibana:${STACK_VERSION} labels: co.elastic.logs/module: kibana volumes: - certs:/usr/share/kibana/config/certs - kibanadata:/usr/share/kibana/data ports: - ${KIBANA_PORT}:5601 environment: - SERVERNAME=kibana - ELASTICSEARCH_HOSTS=https://es01:9200 - ELASTICSEARCH_USERNAME=kibana_system - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD} - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY} - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY} - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY} mem_limit: ${KB_MEM_LIMIT} healthcheck: test: [ "CMD-SHELL", "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'", ] interval: 10s timeout: 10s retries: 120 networks: - elk logstash01: depends_on: es01: condition: service_healthy kibana: condition: service_healthy image: docker.elastic.co/logstash/logstash:${STACK_VERSION} labels: co.elastic.logs/module: logstash user: root volumes: - certs:/usr/share/logstash/certs - logstashdata01:/usr/share/logstash/data - "./observability-configuration/logstash_ingest_data/:/usr/share/logstash/ingest_data/" - "./observability-configuration/elk-stack/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro" environment: - xpack.monitoring.enabled=false - ELASTIC_USER=elastic - ELASTIC_PASSWORD=${ELASTIC_PASSWORD} - ELASTIC_HOSTS=https://es01:9200 networks: - elk apm-server: depends_on: es01: condition: service_healthy image: docker.elastic.co/apm/apm-server:${STACK_VERSION} labels: co.elastic.logs/module: apm-server volumes: - certs:/usr/share/apm-server/config/certs - apmdata:/usr/share/apm-server/data command: > apm-server -e -E output.elasticsearch.hosts="https://es01:9200" -E output.elasticsearch.username=apm_system -E output.elasticsearch.password=${APM_PASSWORD} -E output.elasticsearch.ssl.certificate_authorities=/usr/share/apm-server/config/certs/ca/ca.crt -E apm-server.ssl.enabled=true -E apm-server.ssl.key=/usr/share/apm-server/config/certs/apm-server/apm-server.key -E apm-server.ssl.certificate=/usr/share/apm-server/config/certs/apm-server/apm-server.crt -E apm-server.ssl.certificate_authorities=/usr/share/apm-server/config/certs/ca/ca.crt -E apm-server.kibana.enabled=true -E apm-server.kibana.host="https://kibana:${KIBANA_PORT}" -E logging.level=debug ports: - ${APM_PORT}:8200 healthcheck: test: ["CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://localhost:8200 | grep -q 'build_date'"] interval: 10s timeout: 10s retries: 120 networks: - elk collector: image: otel/opentelemetry-collector-contrib:0.85.0 container_name: collector hostname: collector command: ["--config=/etc/collector-config.yaml"] volumes: - ./collector-config-elk.yaml:/etc/collector-config.yaml - ./observable-spring-boot-app.log:/var/log/observable-spring-boot-app.log # Add the volume for the CA certificate here - certs:/usr/share/apm-server/config/certs depends_on: apm-server: condition: service_healthy kibana: condition: service_healthy ports: - "5555:5555" - "6666:6666" networks: - elk networks: elk: driver: bridge ``` - My collector-config.yaml file is below, ```auto receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:5555 filelog: include: ["/var/log/observable-spring-boot-app.log"] hostmetrics: collection_interval: 10s scrapers: cpu: metrics: system.cpu.time: enabled: true system.cpu.utilization: enabled: true disk: load: filesystem: memory: metrics: system.memory.usage: enabled: true system.memory.utilization: enabled: true network: paging: process: metrics: process.cpu.time: enabled: true process.disk.io: enabled: true process.memory.usage: enabled: true process.memory.virtual: enabled: true process.threads: enabled: true processes: metrics: system.processes.count: enabled: true system.processes.created: enabled: true processors: attributes: actions: - action: insert key: loki.attribute.labels value: log.file.name memory_limiter: check_interval: 1s limit_mib: 2000 batch: timeout: 1s send_batch_size: 1024 exporters: logging: loglevel: warn otlp/elastic: endpoint: apm-server:8200 headers: Authorization: "Bearer ${ELASTIC_APM_SERVER_SECRET}" tls: ca_file: /usr/share/apm-server/config/certs/ca/ca.crt service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging, otlp/elastic] metrics: receivers: [otlp] processors: [batch] exporters: [logging, otlp/elastic] logs: receivers: [otlp] processors: [batch] exporters: [logging, otlp/elastic] ``` Please help me to connect OpenTelemetry with the APM server. I am stuck for almost 1 week. **Steps to reproduce** : 1. Run the docker-compose file to run the elk stack and APM server with a self-signed certificate. 2. Integrate the apm server from Kibana. 3. Use the same collector config file and run the collector service. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 13, 2023, 5:04am UTC](https://discuss.elastic.co/t/cant-connect-opentelemetry-collector-with-apm-server/345064/2 "2023-11-13T05:04:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.