Http: TLS handshake error from x.x.x.x:xxxx EOF

Elastic Observability APM
,
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version: 7.6.2

Elasticsearch version: 7.6.2

APM Server version: 7.6.2

APM Agent language and version: node.js 3.5.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Yes, there's an ALB in front of the APM server, and one infront of the servers hosting the APM agents

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
After configuring APM Server and the node.js agent, I see this in the logs: http: TLS handshake error from <ommitted_IP>:<ommitted_port>: EOF I'm able to see information from the APM Server itself, the agents are trying to send data.

Steps to reproduce:

  1. Install full stack, install apm
  2. Configure apm server according to the docs, to include ssl and token auth (using self signed cert from elasticsearch-certutil)
  3. Configure node.js agent as per docs: https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuring-the-agent.html#agent-configuration-object

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

The only thing in the logs are the http: TLS handshake errors, from each server, on each port. No other errors from any other enabled service eg. auditbeat, metricbeat, kibana, elasticsearh.

2

What's new in APM Server 7.6 is that is supports using TLSv1.3 by default. I'm not a node.js expert but in Java there are some versions where there's a bug in the TLSv1.3 implementation.

Try to remove TLSv1.3 from the supported_protocols in APM Server: https://www.elastic.co/guide/en/apm/server/current/agent-server-ssl.html#_supported_protocols_2

2 Likes
3

Hi @Robert_Bridgeman,
when using self-signed certificates you need to also configure the serverCaCertFile option for the nodejs agent.

4

Thanks I will try that.

5

@simitt is there an option for the client.key and the client.cer as well?

6

I disabled TLS v1.3 and enabled the severcacert function, the error is now gone. I'm not receiving any data from the agents. I do have it set to run before any other service, and still I'm getting no information back.

there's no errors in the logs.

7

Without any error logs from the agent or the server this is hard to figure out. Could you try to set it up in a testing environment without TLS configured to see if there is a general issue with the setup or something specific to TLS?

8

Yes, there's an ALB in front of the APM server

As a first step I'd try to directly connect the APM agent and APM server (without the ALB in between). IMO that's a component we generally don't test with our setup, so there's a chance for issues there. If it still doesn't work without the ALB I'd try to connect without TLS as Silvia suggested.

There are too many components involved right now and the first step should be to isolate the problem.

9

I'll try that

10

It was an issue with TLS on the client side. Cleared it up, it's working now. Thanks!

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.