# Able to connect from pod: but error comes co.elastic.apm.agent.report.IntakeV2ReportingEventHandler

**URL:** <https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587>\
**Category:** APM\
**Tags:** docker, java, server\
**Created:** [December 6, 2022, 1:35pm UTC](https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587 "2022-12-06T13:35:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [December 6, 2022, 1:35pm UTC](https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587/1 "2022-12-06T13:35:07Z")

</div>

Hello APM Server Gurus,

**APM Server version** :  
**[docker.elastic.co/apm/apm-server:7.17.6](http://docker.elastic.co/apm/apm-server:7.17.6)**

**APM Agent language and version** :  
NodeJs : javascript : 3.38 and Java 1.33  
**Browser version** :  
Not applicable  
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:  
Via Operator in K8S cluster

**Fresh install or upgraded from other version?**  
Upgraded using instructions listed [here](https://www.elastic.co/guide/en/cloud-on-k8s/2.4/k8s-upgrading-eck.html)

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Yes. Logstash is being used and sending the output to kafka cluster

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:  
We have installed the apm server in k8s cluster in which our pods are running. The elasticsearch for this apm-server is what we have externalised. This gave us flexibility to point our sandbox/dev/staging kind of k8s cluster to point to same elasticsearch/observability cluster.

As communication between apm-server and pods is 'intra-cluster', we did not opt for SSL verification to avoid unwanted maintenance (our apm-server service end points are not accessible from outside the cluster.

Many a times things work w/o issues i.e. I am able to see java and nodejs services generating data properly and we are able to use the APM in Kibana (vanilla , default w/o any issues).

All the services use configuration like:  
ELASTIC\_APM\_CIRCUIT\_BREAKER\_ENABLED: true  
ELASTIC\_APM\_ENVIRONMENT: sandbox  
ELASTIC\_APM\_SECRET\_TOKEN: xxxxxx-masked-xxxxxxx  
ELASTIC\_APM\_SERVER\_URL: [https://elk-apm-jaeger-apm-http.elastic-apm:8200](https://elk-apm-jaeger-apm-http.elastic-apm:8200)  
ELASTIC\_APM\_TRANSACTION\_SAMPLE\_RATE: 0.1  
**ELASTIC\_APM\_VERIFY\_SERVER\_CERT: false**

When I go to console of my service/pod (debian bulls eye 11) and fire curl -k [https://elk-apm-jaeger-apm-http.elastic-apm:8200](https://elk-apm-jaeger-apm-http.elastic-apm:8200), it does the job saying ignoring verification and removal of -k flag, gives appropriate error message.

```auto
root@testing-c4cadapter-worker-service-78b4585b6-xmbdb:/# curl -k https://elk-apm-jaeger-apm-http.elastic-apm:8200
root@testing-c4cadapter-worker-service-78b4585b6-xmbdb:/# curl https://elk-apm-jaeger-apm-http.elastic-apm:8200
curl: (60) SSL certificate problem: EE certificate key too weak
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

```

On Kibana APM  
I do get the transactions; please note: I get dependencies too... but on console I get below content.

**expectation is**  
_I should not get below content, it generates noise._ Is there any issue from our side?  
I am specifically conveying **to skip SSL verification in configuration**.

I searched the existing messages w/o any much success though.

Problem statement:  
On the pod console, we are getting error saying below:  
**Errors in browser console (if relevant)**:

```auto
2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at https://elk-apm-jaeger-apm-http.elastic-apm:8200/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Default cipher suites: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Supported cipher suites: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: Remote host terminated the handshake

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 9 seconds (+/-10%)

2022-12-05 16:32:28,067 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)

2022-12-05 16:32:28,071 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Unexpected end of file from server, response code is -1

2022-12-05 16:32:28,071 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null

2022-12-05 16:51:32,611 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified

2022-12-05 16:51:32,611 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - elk-apm-jaeger-apm-http.elastic-apm

```

---

<div class="post-metadata">

**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\
**Post date:** [December 7, 2022, 9:35am UTC](https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587/2 "2022-12-07T09:35:26Z")

</div>

Hi @VVK ,

The log is actually generated due to connection issues. Whenever a connectivity problem occurs, the agent prints the TLS information, independently of whether the problem is actually TLS related.

The full log should normally contain exceptions and stacktraces giving you more details on the cause of the connectivity problem.

So, I would suggest you to investigate whether there are any issues with the connection to the APM server. Even though you are receieving data, you still occasionally loose some as indicated by this log message:

```auto
2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error

```

If you want the agent logs to be separated from your application logs, you should configure the agent to log to a file instead:

> **[Logging configuration options | APM Java Agent Reference \[1.x\] | Elastic](https://www.elastic.co/guide/en/apm/agent/java/1.x/config-logging.html#config-log-file)**

---

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [December 7, 2022, 12:39pm UTC](https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587/3 "2022-12-07T12:39:19Z")

</div>

Hello Jonas,

Thanks a ton for quick inputs, let me try.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2022, 8:39am UTC](https://discuss.elastic.co/t/able-to-connect-from-pod-but-error-comes-co-elastic-apm-agent-report-intakev2reportingeventhandler/320587/4 "2022-12-28T08:39:53Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.