How to use Elastic Search CA Cert with OpenTelemetry?

[vsabado (Vlad)]

I downloaded the CA Cert from my Elasticsearch cloud instance and put it in /usr/local/share/ca-certificates/. I then ran update-ca-certificates, which created a .pem file of my certificate under /etc/ssl/certs.

I updated my config.yaml file to connect to Elasticsearch, with the exporter settings below:

exporters:
  logging:
    verbosity: detailed
  otlp/elastic:
    endpoint: https://someid.apm.us-central1.gcp.cloud.es.io:443
    headers:
      Authorization: Bearer someToken
    tls:
      insecure: false
      ca_file: "/etc/ssl/certs/certName.pem"

This is the error that I'm getting directly from the OpenTelemetry log:

authentication handshake failed: x509: certificate signed by unknown authority

However when I establish the connection using curl, the certs match. This the command:

curl -v --cacert "/etc/ssl/certs/certName.pem" https://someid.apm.us-central1.gcp.cloud.es.io:443

The fact that curl works but not the collector really stumps me, so I would appreciate any assistance. Also my collector is a binary that I'm executing inside of a docker container, if that's important. It is also important to note that if I bypassed the tls verification entirely, it works just fine but I need TLS to work in production. Am I not using this cert correctly?

[stephenb (Stephen Brown)]

Hi @vsabado

Are you using our Elastic Cloud service?

If so you should not need to reference a CA at all as all our certificates are signed by a public trusted authority.

This should work with out the CA...where did you get the CA?

curl -v https://someid.apm.us-central1.gcp.cloud.es.io:443

[vsabado (Vlad)]

Hi again @stephenb

I got the CA certificate from Cloud -> Deployments -> MY-DEPLOYMENT -> Security

In there is a download link for a CA Cert, with the description "Use these CA certificates to trust this deployment from an Elastic Cloud Enterprise or self-managed environment"

Yes, I am using your elastic cloud service. One thing to note is that setting insecure to either true or false in my OpenTelemetry config didn't seem to make a difference. It works in either case. Just when I plug in the ca_file does it complain about the CA certificate

So is TLS encryption already enabled without a CA?

[stephenb (Stephen Brown)]

Hi @vsabado

Yes that is the valid CA for your deployment.

Why did you run this

@vsabado

I then ran update-ca-certificates, which created a .pem file of my certificate under /etc/ssl/certs.

The downloaded .crt is in .pem format if I am not mistaken

Did you try the curl as I suggested? You should find that it runs fine without the CA.

curl -v https://someid.apm.us-central1.gcp.cloud.es.io:443

So curious if you actually needs the CA but the docs imply that you do.

According to the docs here

exporters:
  logging:
    verbosity: detailed
  otlp/elastic:
    endpoint: someid.apm.us-central1.gcp.cloud.es.io:443 <--- NOTE no https here
    headers:
      Authorization: Bearer someToken
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/certName.pem

[vsabado (Vlad)]

The downloaded cert is in .crt format. Dropping into my container under /usr/local/share/ca-certificates/ and then running the command update-ca-certificates automatically created a pem file for this cert under /etc/ssl/certs. This directory, from my understanding, is my directory of certs.

Running the curl command turns out just fine, stating that the cert does match. Only when my collector uses that same cert do I get the 509. Not sure why curl works but not the collector in this instance.

Just so I understand, are you suggesting that I remove the https from the endpoint?

Without ca_cert and insecure: true, my collector runs just fine and I can see data flowing up to ES. Not ideal for production though, as we'd prefer the added security if it's necessary

EDIT:

Removing the https in combination with keeping the ca_file still gave me the same issue

[stephenb (Stephen Brown)]

Hmmm.....

Can you show more of the error and not as pasted as an image really hard to debug.

Can you show the entire log? It's cut off so I can't even see it. I want to see the logs leading up to that and the logs after that.

Show before and after... Do you see it's trying to connect to the URL or an IP address?.

I have seen strange issues when people have

/etc/hosts

Files which they have edited and added FQDN and IPs.

Agree, we don't want not secure for production. I'll have to check around and see if I can reproduce this. I'm pretty sure I've sent OTEL data to elastic cloud.

Pretty sure we have some blogs even on that on our site. You might want to take a look.

[vsabado (Vlad)]

Here's the screenshot after I've ran just curl and the endpoint:

This all looks good.

OTel processes the data just fine, but then when it tries to send it to ES after buffering is when we see the issue. I've attached the log of when the error occurs. Removed some sensitive information

<<=== OT ACTUALLY PROCESSING DATA. REMOVED IT FOR SECURITY ===>>
        {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-10-24T20:39:01.724-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING     {"grpc_log": true}
2023-10-24T20:39:01.724-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel picks a new address "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443" to connect {"grpc_log": true}
2023-10-24T20:39:01.724-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {CONNECTING <nil>}     {"grpc_log": true}
2023-10-24T20:39:01.724-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to CONNECTING      {"grpc_log": true}
2023-10-24T20:39:01.818-0500    info    zapgrpc/zapgrpc.go:178  [core] Creating new client transport to "{\n  \"Addr\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"ServerName\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-24T20:39:01.818-0500    warn    zapgrpc/zapgrpc.go:195  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "ServerName": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"       {"grpc_log": true}
2023-10-24T20:39:01.819-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE      {"grpc_log": true}
2023-10-24T20:39:01.819-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}  {"grpc_log": true}
2023-10-24T20:39:01.819-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE       {"grpc_log": true}
2023-10-24T20:39:01.819-0500    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.   {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "6.336685321s"}
2023-10-24T20:39:03.125-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE   {"grpc_log": true}
2023-10-24T20:39:03.125-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {IDLE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}       {"grpc_log": true}
2023-10-24T20:39:03.125-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to IDLE    {"grpc_log": true}
2023-10-24T20:39:08.157-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING     {"grpc_log": true}
2023-10-24T20:39:08.157-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel picks a new address "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443" to connect {"grpc_log": true}
2023-10-24T20:39:08.157-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {CONNECTING <nil>}     {"grpc_log": true}
2023-10-24T20:39:08.157-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to CONNECTING      {"grpc_log": true}
2023-10-24T20:39:08.246-0500    info    zapgrpc/zapgrpc.go:178  [core] Creating new client transport to "{\n  \"Addr\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"ServerName\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-24T20:39:08.246-0500    warn    zapgrpc/zapgrpc.go:195  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "ServerName": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"       {"grpc_log": true}
2023-10-24T20:39:08.246-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE      {"grpc_log": true}
2023-10-24T20:39:08.247-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}  {"grpc_log": true}
2023-10-24T20:39:08.247-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE       {"grpc_log": true}
2023-10-24T20:39:08.247-0500    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.   {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "9.914731973s"}
2023-10-24T20:39:10.743-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE   {"grpc_log": true}
2023-10-24T20:39:10.743-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {IDLE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}       {"grpc_log": true}
2023-10-24T20:39:10.744-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to IDLE    {"grpc_log": true}
2023-10-24T20:39:16.725-0500    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 39}
2023-10-24T20:39:16.727-0500    info    ResourceMetrics #0

<<=== OT ACTUALLY PROCESSING DATA. REMOVED IT FOR SECURITY ===>>

Resource SchemaURL: https://opentelemetry.io/schemas/1.20.0
2023-10-24T20:39:16.732-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING     {"grpc_log": true}
2023-10-24T20:39:16.732-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {CONNECTING <nil>}     {"grpc_log": true}
2023-10-24T20:39:16.732-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to CONNECTING      {"grpc_log": true}
2023-10-24T20:39:16.732-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel picks a new address "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443" to connect {"grpc_log": true}
2023-10-24T20:39:16.835-0500    info    zapgrpc/zapgrpc.go:178  [core] Creating new client transport to "{\n  \"Addr\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"ServerName\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-24T20:39:16.835-0500    warn    zapgrpc/zapgrpc.go:195  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "ServerName": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"       {"grpc_log": true}
2023-10-24T20:39:16.835-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE      {"grpc_log": true}
2023-10-24T20:39:16.835-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}  {"grpc_log": true}
2023-10-24T20:39:16.835-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE       {"grpc_log": true}
2023-10-24T20:39:16.836-0500    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.   {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "4.184529792s"}
2023-10-24T20:39:18.163-0500    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.   {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "7.358993445s"}
2023-10-24T20:39:20.596-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE   {"grpc_log": true}
2023-10-24T20:39:20.596-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {IDLE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}       {"grpc_log": true}
2023-10-24T20:39:20.596-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to IDLE    {"grpc_log": true}
2023-10-24T20:39:21.021-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING     {"grpc_log": true}
2023-10-24T20:39:21.021-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel picks a new address "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443" to connect {"grpc_log": true}
2023-10-24T20:39:21.021-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {CONNECTING <nil>}     {"grpc_log": true}
2023-10-24T20:39:21.022-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to CONNECTING      {"grpc_log": true}
2023-10-24T20:39:21.108-0500    info    zapgrpc/zapgrpc.go:178  [core] Creating new client transport to "{\n  \"Addr\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"ServerName\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-24T20:39:21.108-0500    warn    zapgrpc/zapgrpc.go:195  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "ServerName": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"       {"grpc_log": true}
2023-10-24T20:39:21.108-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE      {"grpc_log": true}
2023-10-24T20:39:21.108-0500    info    zapgrpc/zapgrpc.go:178  [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"}  {"grpc_log": true}
2023-10-24T20:39:21.108-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE       {"grpc_log": true}
2023-10-24T20:39:21.108-0500    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.   {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "4.996221292s"}
^C2023-10-24T20:39:22.213-0500  info    otelcol/collector.go:234        Received signal from OS {"signal": "interrupt"}
2023-10-24T20:39:22.213-0500    info    service/service.go:154  Starting shutdown...
2023-10-24T20:39:22.213-0500    info    service/pipelines.go:130        Stopping receivers...
2023-10-24T20:39:22.213-0500    info    zapgrpc/zapgrpc.go:178  [core] [Server #9 ListenSocket #10] ListenSocket deleted   {"grpc_log": true}
2023-10-24T20:39:22.214-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: closing: EOF     {"grpc_log": true}
2023-10-24T20:39:22.214-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: closing: EOF     {"grpc_log": true}
2023-10-24T20:39:22.214-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client     {"grpc_log": true}
2023-10-24T20:39:22.215-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: closing: EOF     {"grpc_log": true}
2023-10-24T20:39:22.215-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client     {"grpc_log": true}
2023-10-24T20:39:22.216-0500    info    service/pipelines.go:137        Stopping processors...
2023-10-24T20:39:22.216-0500    info    service/pipelines.go:144        Stopping exporters...
2023-10-24T20:39:22.217-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #7] Channel Connectivity change to SHUTDOWN        {"grpc_log": true}
2023-10-24T20:39:22.217-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to SHUTDOWN       {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #7 SubChannel #8] Subchannel deleted       {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #7] Channel deleted     {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #5] Channel Connectivity change to SHUTDOWN        {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to SHUTDOWN       {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #5 SubChannel #6] Subchannel deleted       {"grpc_log": true}
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #5] Channel deleted     {"grpc_log": true}
2023-10-24T20:39:22.219-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel Connectivity change to SHUTDOWN        {"grpc_log": true}
2023-10-24T20:39:22.219-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN       {"grpc_log": true}
2023-10-24T20:39:22.219-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel deleted       {"grpc_log": true}
2023-10-24T20:39:22.219-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel deleted     {"grpc_log": true}
2023-10-24T20:39:22.219-0500    error   exporterhelper/queued_retry.go:175      Exporting failed. No more retries left. Dropping data.     {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "interrupted due to shutdown rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "dropped_items": 73}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:175
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:438
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:136
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:205
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61
2023-10-24T20:39:22.218-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client     {"grpc_log": true}
2023-10-24T20:39:22.219-0500    error   exporterhelper/queued_retry.go:175      Exporting failed. No more retries left. Dropping data.     {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "interrupted due to shutdown rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "dropped_items": 145}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:175
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:438
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:136
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:205
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
        go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61
2023-10-24T20:39:22.220-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: closing: connection error: desc = "error reading from server: read tcp 172.18.0.3:51070->35.227.211.64:443: use of closed network connection"  {"grpc_log": true}
2023-10-24T20:39:22.220-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client     {"grpc_log": true}
2023-10-24T20:39:22.226-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel Connectivity change to SHUTDOWN        {"grpc_log": true}
2023-10-24T20:39:22.227-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN       {"grpc_log": true}
2023-10-24T20:39:22.228-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3 SubChannel #4] Subchannel deleted       {"grpc_log": true}
2023-10-24T20:39:22.230-0500    info    zapgrpc/zapgrpc.go:178  [core] [Channel #3] Channel deleted     {"grpc_log": true}
2023-10-24T20:39:22.231-0500    info    extensions/extensions.go:55     Stopping extensions...
2023-10-24T20:39:22.232-0500    info    service/service.go:168  Shutdown complete.

[stephenb (Stephen Brown)]

Interesting / Weird

Also Again, what version of the Elastic Stack?

And you are not using an edited
/etc/hosts

Here is my complete config that works with Elastic Cloud

Note I ran this on my Mac, I have not on Ubuntu yet... perhaps something different there

./mvnw clean package -Dmaven.test.skip=true

export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://mydepolyment.apm.us-west1.gcp.cloud.es.io:443/
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer safdasdf8OgskmL"
java -javaagent:./opentelemetry-javaagent.jar \
    -jar target/cardatabase-0.0.1-SNAPSHOT.jar

It connects and sends telemetry.... Note no need for CA or anthing

You can try this if you want the repo is

GitHub - bvader/cardatabase: Backend application for cars

The example is setup from the repo to use Elastic Agent ... but you can replace the

./run-with-apm.sh with the commands above... just download the hotel java agent from here

What language of agent are you using?

[vsabado (Vlad)]

I'm getting telemetry data as well. My setup works just fine, at least up until I tried to use the ES cert via ca_file, then it fails to send data up to elastic with that 509 error. It just doesn't seem to like that cert for some reason.

My team wanted to increase our security a bit by including the cert for production. Since the endpoint is already https, do you feel like there's really a need to include the cert for added security?

I'm on v8.5.2

[stephenb (Stephen Brown)]

I just ran this on Ubuntu and it worked as well with no extra settings LTS 20.04.01

$ uname -a
Linux stephenb-es-8-test 5.15.0-1038-gcp #46~20.04.1-Ubuntu SMP Fri Jul 14 09:48:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[stephenb (Stephen Brown)]

@vsabado

I'm getting telemetry data as well. My setup works just fine, at least up until I tried to use the ES cert via ca_file, then it fails to send data up to elastic with that 509 error. It just doesn't seem to like that cert for some reason.

You should NOT need the cert. That is the whole point. It is a Publicly Signed Trusted Cert / CA, not a self-signed cert.

Adding the CA adds little / no value.

I will try adding the CA and see what I get

[vsabado (Vlad)]

Ah I see. I had a hunch that I was not using that cert properly. For added security for the communication that occurs between Elastic Search and the OpenTelemetry collector, what do you recommend?

There is a concern from a team member of mine about someone possibly intercepting the data flow and extracting potentially sensitive data.

Now this is where my lack of knowledge comes in, but is there some sort of encryption that could be done (if it's not already happening) to minimize this risk? We were under the impression that using the CA cert would help us with this but since your last reply, I am not so sure.

To answer a previous question:

I'm just using a config.yaml file to configure my OpenTelemetry collector. We're using the jar you posted earlier to facilitate the automatic instrumentation

[stephenb (Stephen Brown)]

@vsabado

Elastic Search and the OpenTelemetry collector, what do you recommend?

Ok confused Elasticsearch does not communicate with the Opentelemetry Collector it is the other way around?

I think at this point we need to be very precise....

Exactly which components and connections are you wanting to secure.

From this page

Are you talking about the connection I highlighted... If so I would assume it is exactly the same, there is already a publicly signed and trusted cert, no further security setting are needed.

My example was straight from the OTEL Agent to Elasticsearch

I will have to download and start the collector but I would expect it to act the same.

[vsabado (Vlad)]

That is correct, we are looking into increasing security there -- where the data flows up from the OTel collector to Elastic Search. Sorry I should've been more clear but the direction there.

Could you explain a little bit about what occurs there in terms of data encryption, if there are any? That's the bit that I don't know about but would need to be able to explain to the rest of the team

Edit:

I don't think I fully understand why there is no need for the CA Cert here. Could you please elaborate a little bit on that

[stephenb (Stephen Brown)]

@vsabado

I don't think I fully understand why there is no need for the CA Cert here. Could you please elaborate a little bit on that

I think you should learn a bit more about certificates and certificate validation if you do not really understand these concepts I suggest perhaps you do a little reading on the subject.

Here is my total collector / exporter config

It connects securely to Elastic Eloud APM server.

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
    verbosity: detailed
  otlp:
    endpoint: mydempolyment.apm.us-west1.gcp.cloud.es.io:443
    headers:
      Authorization: "Bearer sdfsfsgfx7o8OgskmL"
    tls:
      insecure: false # Means that TLS is used, and certificate must be valid

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Interesting As soon as I try to add the ca_file

    tls:
      insecure: false
      ca_file: ./E8KJSHKDLSJHSFLKJFHSDLFKJ.crt

I get the same error you do... in IMHO This should work... or I do not understand something about this connection... which is possible but I work with certs and CAs a fair amount.

This does seem odd / wrong... and is perhaps a bug on that OTEL side, I do NOT think this an error on the Elastic Cloud Side.
I see some similar bugs / issue filed against OTEL Exported

In short, Elastic Cloud has publicly Signed and Trusted Certs, your OS would reject them if they were not, OS typically do NOT honor self-signed / no trusted certs

From the OTEL Docs here

ca_file : Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA.

So that means that when ca_file is empty it validate with the system root CA, only valid publicly Signed Certs with with Trussted CAs (or CAs your org add to the System) will [pass the validation

[stephenb (Stephen Brown)]

Interesting Solved... so what ca_file wants is Full Chain Cert (which is not clear what they are asking for normal a CA file is JUST the CA) .. so IMHO OTEL should specific that it wants a full chain certificate or this is some sort of bug

I got it to work.

So what I did is use Chrome and downloaded the full cert then set it as the ca_file and it connected ... I am not sure this adds any additional security.

then I set the cert

    tls:
      insecure: false
      ca_file: test.cer

and the collector connected without the error!

[vsabado (Vlad)]

Elated to see that you solved it! Trying it out on my end now.

What did you choose here when you exported the cert?

This seems like an entirely different cert from we've downloaded from the security page. Is that right?

[stephenb (Stephen Brown)]

I am not sure what browser you are using I used Chrome no option ... you will need to try.

@vsabado

This seems like an entirely different cert from we've downloaded from the security page. Is that right?

Again I think you need to learn a bit more about certs and even I am only Mid/High Level not a full expert.

What you downloaded from the Security page is a CA (a Certificate Authority, What CA the Server Certs were used to be created) which is different from a full chained certificate that is presented by the server (what is in your browser)

[vsabado (Vlad)]

I'm also on chrome. I'll try out the options and see which one produces the cert.

Just wanted to know if the cert downloaded from the security page is the same as the one downloaded from the browser. They don't seem to be

[stephenb (Stephen Brown)]

Ohh. the first one...