} {"grpc_log": true}
2023-10-24T20:39:21.022-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3] Channel Connectivity change to CONNECTING {"grpc_log": true}
2023-10-24T20:39:21.108-0500 info zapgrpc/zapgrpc.go:178 [core] Creating new client transport to "{\n \"Addr\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n \"ServerName\": \"ESEndpoint.apm.us-central1.gcp.cloud.es.io:443\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority" {"grpc_log": true}
2023-10-24T20:39:21.108-0500 warn zapgrpc/zapgrpc.go:195 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
"ServerName": "ESEndpoint.apm.us-central1.gcp.cloud.es.io:443",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority" {"grpc_log": true}
2023-10-24T20:39:21.108-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE {"grpc_log": true}
2023-10-24T20:39:21.108-0500 info zapgrpc/zapgrpc.go:178 [core] pickfirstBalancer: UpdateSubConnState: 0xc0014821f8, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"} {"grpc_log": true}
2023-10-24T20:39:21.108-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE {"grpc_log": true}
2023-10-24T20:39:21.108-0500 info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "interval": "4.996221292s"}
^C2023-10-24T20:39:22.213-0500 info otelcol/collector.go:234 Received signal from OS {"signal": "interrupt"}
2023-10-24T20:39:22.213-0500 info service/service.go:154 Starting shutdown...
2023-10-24T20:39:22.213-0500 info service/pipelines.go:130 Stopping receivers...
2023-10-24T20:39:22.213-0500 info zapgrpc/zapgrpc.go:178 [core] [Server #9 ListenSocket #10] ListenSocket deleted {"grpc_log": true}
2023-10-24T20:39:22.214-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: closing: EOF {"grpc_log": true}
2023-10-24T20:39:22.214-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: closing: EOF {"grpc_log": true}
2023-10-24T20:39:22.214-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client {"grpc_log": true}
2023-10-24T20:39:22.215-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: closing: EOF {"grpc_log": true}
2023-10-24T20:39:22.215-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client {"grpc_log": true}
2023-10-24T20:39:22.216-0500 info service/pipelines.go:137 Stopping processors...
2023-10-24T20:39:22.216-0500 info service/pipelines.go:144 Stopping exporters...
2023-10-24T20:39:22.217-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #7] Channel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.217-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #7 SubChannel #8] Subchannel deleted {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #7] Channel deleted {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #5] Channel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #5 SubChannel #6] Subchannel deleted {"grpc_log": true}
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #5] Channel deleted {"grpc_log": true}
2023-10-24T20:39:22.219-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.219-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.219-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel deleted {"grpc_log": true}
2023-10-24T20:39:22.219-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel deleted {"grpc_log": true}
2023-10-24T20:39:22.219-0500 error exporterhelper/queued_retry.go:175 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "interrupted due to shutdown rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "dropped_items": 73}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:175
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:438
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:136
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:205
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61
2023-10-24T20:39:22.218-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client {"grpc_log": true}
2023-10-24T20:39:22.219-0500 error exporterhelper/queued_retry.go:175 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "interrupted due to shutdown rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"", "dropped_items": 145}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:175
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:438
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:136
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:205
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61
2023-10-24T20:39:22.220-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: closing: connection error: desc = "error reading from server: read tcp 172.18.0.3:51070->35.227.211.64:443: use of closed network connection" {"grpc_log": true}
2023-10-24T20:39:22.220-0500 info zapgrpc/zapgrpc.go:178 [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client {"grpc_log": true}
2023-10-24T20:39:22.226-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3] Channel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.227-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN {"grpc_log": true}
2023-10-24T20:39:22.228-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3 SubChannel #4] Subchannel deleted {"grpc_log": true}
2023-10-24T20:39:22.230-0500 info zapgrpc/zapgrpc.go:178 [core] [Channel #3] Channel deleted {"grpc_log": true}
2023-10-24T20:39:22.231-0500 info extensions/extensions.go:55 Stopping extensions...
2023-10-24T20:39:22.232-0500 info service/service.go:168 Shutdown complete.
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 2:27am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/8 "2023-10-25T02:27:24Z")
Interesting / Weird
Also Again, what version of the Elastic Stack?
And you are not using an edited
`/etc/hosts`
Here is my complete config that works with Elastic Cloud
Note I ran this on my Mac, I have not on Ubuntu yet... perhaps something different there
```auto
./mvnw clean package -Dmaven.test.skip=true
export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://mydepolyment.apm.us-west1.gcp.cloud.es.io:443/
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer safdasdf8OgskmL"
java -javaagent:./opentelemetry-javaagent.jar \
-jar target/cardatabase-0.0.1-SNAPSHOT.jar
```
It connects and sends telemetry.... Note no need for CA or anthing
You can try this if you want the repo is
> **[GitHub - bvader/cardatabase: Backend application for cars](https://github.com/bvader/cardatabase)**
>
> Backend application for cars. Contribute to bvader/cardatabase development by creating an account on GitHub.
The example is setup from the repo to use Elastic Agent ... but you can replace the
`./run-with-apm.sh` with the commands above... just download the hotel java agent from [here](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar)
What language of agent are you using?
---
### Author:  [@vsabado](https://discuss.elastic.co/u/vsabado)
#### Post date: [October 25, 2023, 3:21am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/9 "2023-10-25T03:21:14Z")
I'm getting telemetry data as well. My setup works just fine, at least up until I tried to use the ES cert via ca\_file, then it fails to send data up to elastic with that 509 error. It just doesn't seem to like that cert for some reason.
My team wanted to increase our security a bit by including the cert for production. Since the endpoint is already https, do you feel like there's really a need to include the cert for added security?
I'm on v8.5.2
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 3:21am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/10 "2023-10-25T03:21:29Z")
I just ran this on Ubuntu and it worked as well with no extra settings LTS 20.04.01
```auto
$ uname -a
Linux stephenb-es-8-test 5.15.0-1038-gcp #46~20.04.1-Ubuntu SMP Fri Jul 14 09:48:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 3:23am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/11 "2023-10-25T03:23:41Z")
> [@vsabado](#):
>
> I'm getting telemetry data as well. My setup works just fine, at least up until I tried to use the ES cert via ca\_file, then it fails to send data up to elastic with that 509 error. It just doesn't seem to like that cert for some reason.
You should NOT need the cert. That is the whole point. It is a Publicly Signed Trusted Cert / CA, not a self-signed cert.
Adding the CA adds little / no value.
I will try adding the CA and see what I get
---
### Author:  [@vsabado](https://discuss.elastic.co/u/vsabado)
#### Post date: [October 25, 2023, 3:28am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/12 "2023-10-25T03:28:52Z")
Ah I see. I had a hunch that I was not using that cert properly. For added security for the communication that occurs between Elastic Search and the OpenTelemetry collector, what do you recommend?
There is a concern from a team member of mine about someone possibly intercepting the data flow and extracting potentially sensitive data.
Now this is where my lack of knowledge comes in, but is there some sort of encryption that could be done (if it's not already happening) to minimize this risk? We were under the impression that using the CA cert would help us with this but since your last reply, I am not so sure.
To answer a previous question:
I'm just using a config.yaml file to configure my OpenTelemetry collector. We're using the jar you posted earlier to facilitate the automatic instrumentation
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 3:34am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/13 "2023-10-25T03:34:58Z")
> [@vsabado](#):
>
> Elastic Search and the OpenTelemetry collector, what do you recommend?
Ok confused Elasticsearch does not communicate with the Opentelemetry Collector it is the other way around?
I think at this point we need to be very precise....
Exactly which components and connections are you wanting to secure.
[From this page](https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html)
Are you talking about the connection I highlighted... If so I would assume it is exactly the same, there is already a publicly signed and trusted cert, no further security setting are needed.
My example was straight from the OTEL Agent to Elasticsearch
I will have to download and start the collector but I would expect it to act the same.
---
### Author:  [@vsabado](https://discuss.elastic.co/u/vsabado)
#### Post date: [October 25, 2023, 3:37am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/14 "2023-10-25T03:37:59Z")
That is correct, we are looking into increasing security there -- where the data flows up from the OTel collector to Elastic Search. Sorry I should've been more clear but the direction there.
Could you explain a little bit about what occurs there in terms of data encryption, if there are any? That's the bit that I don't know about but would need to be able to explain to the rest of the team
Edit:
I don't think I fully understand why there is no need for the CA Cert here. Could you please elaborate a little bit on that
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 4:22am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/15 "2023-10-25T04:22:09Z")
> [@vsabado](#):
>
> I don't think I fully understand why there is no need for the CA Cert here. Could you please elaborate a little bit on that
I think you should learn a bit more about certificates and certificate validation if you do not really understand these concepts I suggest perhaps you do a little reading on the subject.
Here is my total collector / exporter config
It connects securely to Elastic Eloud APM server.
```auto
receivers:
otlp:
protocols:
grpc:
exporters:
logging:
verbosity: detailed
otlp:
endpoint: mydempolyment.apm.us-west1.gcp.cloud.es.io:443
headers:
Authorization: "Bearer sdfsfsgfx7o8OgskmL"
tls:
insecure: false # Means that TLS is used, and certificate must be valid
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp]
```
Interesting As soon as I try to add the `ca_file`
```auto
tls:
insecure: false
ca_file: ./E8KJSHKDLSJHSFLKJFHSDLFKJ.crt
```
I get the same error you do... in IMHO This should work... or I do not understand something about this connection... which is possible but I work with certs and CAs a fair amount.
This does seem odd / wrong... and is perhaps a bug on that OTEL side, I do NOT think this an error on the Elastic Cloud Side.
I see some similar bugs / issue filed against OTEL Exported
In short, Elastic Cloud has publicly Signed and Trusted Certs, your OS would reject them if they were not, OS typically do NOT honor self-signed / no trusted certs
From the OTEL Docs [here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
> `ca_file` : Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA.
So that means that when `ca_file` is empty it validate with the system root CA, only valid publicly Signed Certs with with Trussted CAs (or CAs your org add to the System) will [pass the validation
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 4:43am UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/16 "2023-10-25T04:43:09Z")
Interesting **Solved**... so what `ca_file` wants is Full Chain Cert (which is not clear what they are asking for normal a CA file is JUST the CA) .. so IMHO OTEL should specific that it wants a full chain certificate or this is some sort of bug
I got it to work.
So what I did is use Chrome and downloaded the full cert then set it as the `ca_file` and it connected ... I am not sure this adds any additional security.
then I set the cert
```auto
tls:
insecure: false
ca_file: test.cer
```
and the collector connected without the error!
---
### Author:  [@vsabado](https://discuss.elastic.co/u/vsabado)
#### Post date: [October 25, 2023, 2:09pm UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/17 "2023-10-25T14:09:10Z")
Elated to see that you solved it! Trying it out on my end now.
What did you choose here when you exported the cert?
This seems like an entirely different cert from we've downloaded from the security page. Is that right?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 2:58pm UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/18 "2023-10-25T14:58:08Z")
I am not sure what browser you are using I used Chrome no option ... you will need to try.
> [@vsabado](#):
>
> This seems like an entirely different cert from we've downloaded from the security page. Is that right?
Again I think you need to learn a bit more about certs and even I am only Mid/High Level not a full expert.
What you downloaded from the Security page is a CA (a Certificate Authority, What CA the Server Certs were used to be created) which is different from a full chained certificate that is presented by the server (what is in your browser)
---
### Author:  [@vsabado](https://discuss.elastic.co/u/vsabado)
#### Post date: [October 25, 2023, 3:20pm UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/19 "2023-10-25T15:20:42Z")
I'm also on chrome. I'll try out the options and see which one produces the cert.
Just wanted to know if the cert downloaded from the security page is the same as the one downloaded from the browser. They don't seem to be
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 25, 2023, 3:22pm UTC](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667/20 "2023-10-25T15:22:51Z")
Ohh. the first one...
[Next page](https://discuss.elastic.co/t/how-to-use-elastic-search-ca-cert-with-opentelemetry/345667.md?page=2)