# APM Server Not Receiving gRPC traffic but is receiving https traffic **URL:** \ **Category:** APM\ **Tags:** docker, server\ **Created:** [July 1, 2021, 6:57pm UTC](https://discuss.elastic.co/t/apm-server-not-receiving-grpc-traffic-but-is-receiving-https-traffic/277590 "2021-07-01T18:57:12Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [July 1, 2021, 6:57pm UTC](https://discuss.elastic.co/t/apm-server-not-receiving-grpc-traffic-but-is-receiving-https-traffic/277590/1 "2021-07-01T18:57:12Z")
**Kibana version** : 7.13.2 **Elasticsearch version** : 7.13.2 **APM Server version** : 7.13.2 **APM Agent language and version** : [Golang OTEL v1.0.0-RC1](https://github.com/open-telemetry/opentelemetry-go) **Browser version** : N/A **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: ECK (Operator 1.6.0) ```auto --- apiVersion: apm.k8s.elastic.co/v1 kind: ApmServer metadata: name: apm-prod namespace: apm-prod spec: version: 7.13.2 count: 1 elasticsearchRef: name: es-prod namespace: elastic-prod kibanaRef: name: kibana-prod namespace: kibana-prod http: tls: certificate: secretName: apm-cert config: apm-server: auth: api_key: enabled: true limit: 100 capture_personal_data: true # TODO check if we want this enabled # TODO RUM? kibana: enabled: true ssl.enabled: true ssl.verification_mode: certificate ssl.certificate_authorities: ["/usr/share/apm-server/certs/"] output: elasticsearch: ssl.enabled: true ssl.certificate_authorities: ["/usr/share/apm-server/certs/"] ssl.verification_mode: certificate http: enabled: true host: 0.0.0.0 port: 5067 monitoring: enabled: false cluster_uuid: "" podTemplate: metadata: annotations: linkerd.io/inject: enabled config.linkerd.io/proxy-cpu-limit: "2" co.elastic.metrics/raw: '[{"enabled":true,"module":"beat","hosts":["http://${data.host}:5067"],"metricsets":["stats","state"],"period":"10s","timeout":"3s","xpack":{"enabled":true}}]' spec: automountServiceAccountToken: true containers: - name: apm-server resources: limits: memory: 2Gi cpu: 2 volumeMounts: - name: mountPath: /usr/share/apm-server/certs volumes: - name: secret: secretName: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: elasticsearch.k8s.elastic.co/cluster-name: es-prod topologyKey: kubernetes.io/hostname ``` **Fresh install or upgraded from other version?** Fresh Install **Is there anything special in your setup?** Kubernetes Node Port -\> APM Server -\> Elasticsearch **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I have deployed the APM server and am attempting to use the new [OTEL integration](https://www.elastic.co/guide/en/apm/get-started/current/open-telemetry-elastic.html#open-telemetry-elastic-protocol) with my code, however, it doesn't seem like the gRPC is working correctly. If I send a curl request to http I see the log in the APM server, example: 1. `curl -X GET -k https://: -H "authorization=ApiKey " -vvv` 2. Get a 200 response: ```auto Note: Unnecessary use of -X or --request, GET is already inferred. * Trying 192.168.247.1:31759... * TCP_NODELAY set * Connected to (192.168.247.1) port (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x556c968ed990) > GET / HTTP/2 > Host: : > User-Agent: curl/7.66.0 > Accept: */* > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)! * Connection state changed (MAX_CONCURRENT_STREAMS == 250)! < HTTP/2 200 < x-content-type-options: nosniff < content-length: 0 < date: Thu, 01 Jul 2021 18:40:46 GMT < * Connection #0 to host left intact ``` 1. Log appears in APM server: ```auto {"log.level":"info","@timestamp":"2021-07-01T18:52:42.472Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.66.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"f1b2d5a1-fc1f-4d75-88c0-2dab24d3a204","event.duration":150808,"http.response.status_code":200,"ecs.version":"1.6.0" ``` If I however send a [grpcurl](https://github.com/fullstorydev/grpcurl) request with the same info: 1. `docker run fullstorydev/grpcurl -insecure -H 'authorization:ApiKey : list` 2. I get back the following error: ```auto Failed to list services: server does not support the reflection API ``` 1. I don't see any log in the APM server showing that a gRPC request even made it to the server. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [July 2, 2021, 3:01pm UTC](https://discuss.elastic.co/t/apm-server-not-receiving-grpc-traffic-but-is-receiving-https-traffic/277590/2 "2021-07-02T15:01:04Z")
This turned out to be an issue on my end, where I was using gRPC with TLS, but wasn't properly setting the Root CA on the gRPC client config. After setting the root CA, it was able to properly connect. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 23, 2021, 11:01am UTC](https://discuss.elastic.co/t/apm-server-not-receiving-grpc-traffic-but-is-receiving-https-traffic/277590/3 "2021-07-23T11:01:15Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.