APM Server Standalone - TLS handshake error from

Elastic Observability APM
1

Kibana version: 8.18.0

Elasticsearch version: 8.18.0

APM Server version: 8.18.0

Hello , i have a elastic cluster composed by 3 nodes with all roles. A kibana node , and another vm with Apm-Server installed.

I have an openshift cluster where we deployed the EDOT Collector. And we instrument an application to collect the traces.

I don't have access to internet so i cannot use the Elastic Fleet Agent. So i install the apm-server with RPM on my vm.

This is the conf :

apm-server:
  host: "0.0.0.0:8200"  # APM Server che ascolta sull'IP
  auth:
    secret_token: "----------------------------------------------------------"

  ssl:
    enabled: true
    certificate: "/etc/apm-server/certs/apm-server.crt"  # Selft-signed Certificate
    key: "/etc/apm-server/certs/apm-server.key"  # Private Key
    verification_mode: none

output.elasticsearch:
  hosts: ["https://elk.logs-rm.collaudo.it:9200"]  # Elastic VIP-Load Balancer
  protocol: "https"
  username: "elastic"
  password: "--------------------------------"
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/etc/apm-server/certs/global-ca.cer"]   #  Concatenated CA  (My company CA(for elastic cluster) + Self Signed (for apm-server)
  ssl.certificate: "/etc/apm-server/certs/elk.logs-rm.collaudo.it.cer"
  ssl.key: "/etc/apm-server/certs/wildcardcert-coll-logs.key"

[root@elk-coll-apm-rm-01 apm-server]# netstat -tulnp | grep 8200 tcp6 0 0 :::8200 :::* LISTEN 92397/apm-server

If i check logs i see

Jun 06 12:37:48 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92397]: {"log.level":"error","@timestamp":"2025-06-06T12:37:48.272+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from  10.6.228.131:21701: EOF","service.name":"apm-server","ecs.version":"1.6.0"}

The IP in the logs is the IP of the VIP - LoadBalancer we configure with a DNS of "https://elk.apm.logs-rm.collaudo.it" on port 8200.

What can i do? If i go on Observability Kibana , and i do install apm-agent , it doesn't work.

help me plz

2

Hello @Andex ,

Welcome back.

To troubleshoot this i will try to run the curl from the VM to elasticsearch VIP & try to fix this issue.

curl -k -u elastic:password https://elk.logs-rm.collaudo.it:9200

Could you please see if this is going through if not what is the error & try to fix the connectivity between APM => Elasticnodes (VIP URL)

Thanks!!

3

Hi @Tortoise

From the Vm with APM Server installed to Elasticsearch cluster works

Enter host password for user 'elastic':
{
  "name" : "elk-coll-logs-rm-01",
  "cluster_name" : "elk-coll-logs-rm",
  "cluster_uuid" : "VkfGcD__RG2jzBxWIelQVg",
  "version" : {
    "number" : "8.18.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "04e979aa50b657bebd4a0937389308de82c2bdad",
    "build_date" : "2025-04-10T10:09:16.444104780Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
4

Thanks @Andex for sharing the details.

As per the conf shared is see below :

ssl.verification_mode: none

Could you please try to update to full or certificate & see because if it is none :

Performs no verification of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Thanks!!

5

@Tortoise Ok same thing with full and certificate

I provide you more logs

Jun 06 13:46:21 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"info","@timestamp":"2025-06-06T13:46:21.500+0200","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start","file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:21 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"info","@timestamp":"2025-06-06T13:46:21.500+0200","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start","file.name":"beater/http.go","file.line":98},"message":"SSL enabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:21 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:46:21.500+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from [::1]:37912: client sent an HTTP request to an HTTPS server","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:21 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"info","@timestamp":"2025-06-06T13:46:21.510+0200","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":68},"message":"no longer blocking ingestion as all precondition checks are now satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:28 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:46:28.584+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22113: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:38 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:46:38.585+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22114: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:48 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:46:48.586+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22115: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:46:58 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:46:58.587+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22116: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:47:08 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:47:08.587+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22117: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:47:18 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:47:18.588+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22118: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
Jun 06 13:47:28 elk-coll-apm-rm-01.logs-rm.collaudo.it apm-server[92602]: {"log.level":"error","@timestamp":"2025-06-06T13:47:28.589+0200","log.logger":"beater.http","log.origin":{"function":"net/http.(*Server).logf","file.name":"http/server.go","file.line":3632},"message":"http: TLS handshake error from 10.6.228.131:22119: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
(END)