APM Server Not Receiving gRPC traffic but is receiving https traffic

[BenB196]

Kibana version:
7.13.2
Elasticsearch version:
7.13.2
APM Server version:
7.13.2
APM Agent language and version:
Golang OTEL v1.0.0-RC1
Browser version:
N/A
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
ECK (Operator 1.6.0)

---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-prod
  namespace: apm-prod
spec:
  version: 7.13.2
  count: 1
  elasticsearchRef:
    name: es-prod
    namespace: elastic-prod
  kibanaRef:
    name: kibana-prod
    namespace: kibana-prod
  http:
    tls:
      certificate:
        secretName: apm-cert
  config:
    apm-server:
      auth:
        api_key:
          enabled: true
          limit: 100
      capture_personal_data: true # TODO check if we want this enabled
      # TODO RUM?
      kibana:
        enabled: true
        ssl.enabled: true
        ssl.verification_mode: certificate
        ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
    output:
      elasticsearch:
        ssl.enabled: true
        ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
        ssl.verification_mode: certificate
    http:
      enabled: true
      host: 0.0.0.0
      port: 5067
    monitoring:
      enabled: false
      cluster_uuid: "<snipped>"
  podTemplate:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/proxy-cpu-limit: "2"
        co.elastic.metrics/raw: '[{"enabled":true,"module":"beat","hosts":["http://${data.host}:5067"],"metricsets":["stats","state"],"period":"10s","timeout":"3s","xpack":{"enabled":true}}]'
    spec:
      automountServiceAccountToken: true
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 2Gi
              cpu: 2
          volumeMounts:
            - name: <snipped>
              mountPath: /usr/share/apm-server/certs
      volumes:
        - name: <snipped>
          secret:
            secretName: <snipped>
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    elasticsearch.k8s.elastic.co/cluster-name: es-prod
                topologyKey: kubernetes.io/hostname

Fresh install or upgraded from other version?
Fresh Install

Is there anything special in your setup?

Kubernetes Node Port -> APM Server -> Elasticsearch

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have deployed the APM server and am attempting to use the new OTEL integration with my code, however, it doesn't seem like the gRPC is working correctly.

If I send a curl request to http I see the log in the APM server, example:

1. curl -X GET -k https://<server_name>:<node_port> -H "authorization=ApiKey <base64 encoded API ID:Key>" -vvv
2. Get a 200 response:

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.247.1:31759...
* TCP_NODELAY set
* Connected to <server_name> (192.168.247.1) port <node_port> (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
<snipped cert info>
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x556c968ed990)
> GET / HTTP/2
> Host: <server_name>:<node_port>
> User-Agent: curl/7.66.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< x-content-type-options: nosniff
< content-length: 0
< date: Thu, 01 Jul 2021 18:40:46 GMT
<
* Connection #0 to host <server_name> left intact

3. Log appears in APM server:

{"log.level":"info","@timestamp":"2021-07-01T18:52:42.472Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.66.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"f1b2d5a1-fc1f-4d75-88c0-2dab24d3a204","event.duration":150808,"http.response.status_code":200,"ecs.version":"1.6.0"

If I however send a grpcurl request with the same info:

1. docker run fullstorydev/grpcurl -insecure -H 'authorization:ApiKey <base64 encoded API ID:Key' <server_name>:<node_port> list
2. I get back the following error:

Failed to list services: server does not support the reflection API

3. I don't see any log in the APM server showing that a gRPC request even made it to the server.

[BenB196]

This turned out to be an issue on my end, where I was using gRPC with TLS, but wasn't properly setting the Root CA on the gRPC client config. After setting the root CA, it was able to properly connect.