Able to connect from pod: but error comes co.elastic.apm.agent.report.IntakeV2ReportingEventHandler

Hello APM Server Gurus,

APM Server version:
docker.elastic.co/apm/apm-server:7.17.6

APM Agent language and version:
NodeJs : javascript : 3.38 and Java 1.33
Browser version:
Not applicable
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Via Operator in K8S cluster

Fresh install or upgraded from other version?
Upgraded using instructions listed here

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Yes. Logstash is being used and sending the output to kafka cluster

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We have installed the apm server in k8s cluster in which our pods are running. The elasticsearch for this apm-server is what we have externalised. This gave us flexibility to point our sandbox/dev/staging kind of k8s cluster to point to same elasticsearch/observability cluster.

As communication between apm-server and pods is 'intra-cluster', we did not opt for SSL verification to avoid unwanted maintenance (our apm-server service end points are not accessible from outside the cluster.

Many a times things work w/o issues i.e. I am able to see java and nodejs services generating data properly and we are able to use the APM in Kibana (vanilla , default w/o any issues).

All the services use configuration like:
ELASTIC_APM_CIRCUIT_BREAKER_ENABLED: true
ELASTIC_APM_ENVIRONMENT: sandbox
ELASTIC_APM_SECRET_TOKEN: xxxxxx-masked-xxxxxxx
ELASTIC_APM_SERVER_URL: https://elk-apm-jaeger-apm-http.elastic-apm:8200
ELASTIC_APM_TRANSACTION_SAMPLE_RATE: 0.1
ELASTIC_APM_VERIFY_SERVER_CERT: false

When I go to console of my service/pod (debian bulls eye 11) and fire curl -k https://elk-apm-jaeger-apm-http.elastic-apm:8200, it does the job saying ignoring verification and removal of -k flag, gives appropriate error message.

root@testing-c4cadapter-worker-service-78b4585b6-xmbdb:/# curl -k https://elk-apm-jaeger-apm-http.elastic-apm:8200
root@testing-c4cadapter-worker-service-78b4585b6-xmbdb:/# curl https://elk-apm-jaeger-apm-http.elastic-apm:8200
curl: (60) SSL certificate problem: EE certificate key too weak
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

On Kibana APM
I do get the transactions; please note: I get dependencies too... but on console I get below content.

expectation is
I should not get below content, it generates noise. Is there any issue from our side?
I am specifically conveying to skip SSL verification in configuration.

I searched the existing messages w/o any much success though.

Problem statement:
On the pod console, we are getting error saying below:
Errors in browser console (if relevant):

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at https://elk-apm-jaeger-apm-http.elastic-apm:8200/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Default cipher suites: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Supported cipher suites: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: Remote host terminated the handshake

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 9 seconds (+/-10%)

2022-12-05 16:32:28,067 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)

2022-12-05 16:32:28,071 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Unexpected end of file from server, response code is -1

2022-12-05 16:32:28,071 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null

2022-12-05 16:51:32,611 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified

2022-12-05 16:51:32,611 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null

2022-12-05 16:51:32,612 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - elk-apm-jaeger-apm-http.elastic-apm

Hi @VVK ,

The log is actually generated due to connection issues. Whenever a connectivity problem occurs, the agent prints the TLS information, independently of whether the problem is actually TLS related.

The full log should normally contain exceptions and stacktraces giving you more details on the cause of the connectivity problem.

So, I would suggest you to investigate whether there are any issues with the connection to the APM server. Even though you are receieving data, you still occasionally loose some as indicated by this log message:

2022-12-05 13:19:59,063 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error

If you want the agent logs to be separated from your application logs, you should configure the agent to log to a file instead:

Hello Jonas,

Thanks a ton for quick inputs, let me try.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.